Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- KX Community
- Discussion Forums
- KX Tech
- Re: Timing/Memory surprise on 3 versions of <Condi...

Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

Options

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2 weeks ago

Hi,

A bit puzzled here... Say I have a table comprised of 2 columns, with the first one made of variable length lists, and the second showing a binary:

t:([] a:{x?50} each 2+100000?100; b:100000?0b)

I want to apply a specific function to each list found in t`a, depending on the binary t`b. In this example, let's simply use <first> and <last> as dependant functions.

Below are 3 possible syntax versions and their time/memory consumption. I would have expected the <scan> version to beat the 2 others by a substantial factor, at least on timing (with +/- impact on memory). The reality seems to be the opposite:

Cond$ with Each

q)\ts {$[x`b; last x`a; first x`a]} each t

40 3697968

Cond? with Each within function

q)\ts {?[x`b; last each x`a; first each x`a]}t

16 4746672

Scan

q)\ts ({?[y`b; last; first] y`a}\)[::;t]

62 12746416

Is there an issue with my wording of the scan version (4x slower, 2.5x memory hungry), or is the second one the actual optimal solution? As usual, your insight is highly appreciated. Thx.

Solved! Go to Solution.

1 ACCEPTED SOLUTION

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2 weeks ago - last edited 2 weeks ago

This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable 'x' inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.

Your second version is fastest as it operates on 'x`b' as a vector rather than inside 'each'.

One other possible variation is shown below:

```
q)\ts {?[x`b; last each x`a; first each x`a]}t
16 4746672
q)\ts {((first;last) x`b)@'x`a}t
7 4746640
```

It's goal is to avoid calculating 'last each' and 'first each' for every row.

Instead it uses each both (') to apply first or last after it is known which function is needed.

2 REPLIES 2

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2 weeks ago - last edited 2 weeks ago

This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable 'x' inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.

Your second version is fastest as it operates on 'x`b' as a vector rather than inside 'each'.

One other possible variation is shown below:

```
q)\ts {?[x`b; last each x`a; first each x`a]}t
16 4746672
q)\ts {((first;last) x`b)@'x`a}t
7 4746640
```

It's goal is to avoid calculating 'last each' and 'first each' for every row.

Instead it uses each both (') to apply first or last after it is known which function is needed.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2 weeks ago

Thank you for both the explanation and the more efficient <each both> alternative... Best.

Related Content

- Timing/Memory surprise on 3 versions of <Conditional function applied to lists> in KX Technology
- embedPy: q list imported as numpy array in KX Technology
- C++ convert large decimal to KDB in New kdb+/q users question forum
- SQL query to Functional query[Parse Tree] in New kdb+/q users question forum
- Lists, dictionaries, tables and lists of dictionaries in KX Technology

Main Office Contacts

**EMEA**

Tel: +44 (0)28 3025 2242

**AMERICAS**

Tel: +1 (212) 447 6700

**APAC**

Tel: +61 (0)2 9236 5700

Useful Information

Resources

Popular Links

Follow Us

KX. All Rights Reserved.

KX and kdb+ are registered trademarks of KX Systems, Inc., a subsidiary of FD Technologies plc.