Profiling Q Functions
Why Q Tips?
- Unique perspective
- Biggest impact
- Show. Don’t tell
Q Tips CEP
- Running the CEP
$ q cep.q -p 5001 -w 200 -t 1000 -eod 16:00 -db qdb
- Parameters
q)p
| ()
ref | `:ref.csv
eod | 0D16:00:00.000000000
db | `:qdb
debug| 0b
log | 2
Tables
q)price
id| px time
--| --------------------------------------
0 | 110.1492 2015.07.15D10:06:04.631217000
1 | 80.12466 2015.07.15D10:06:04.631217000
2 | 10.02082 2015.07.15D10:06:04.631217000
3 | 4.986975 2015.07.15D10:06:04.631217000
4 | 1.957649 2015.07.15D10:06:04.631217000
q)quote
id| bs bp ap as time
--| --------------------------------------------------
0 | 6 110.1 110.15 23 2015.07.15D10:06:04.631217000
1 | 7 80.1 80.15 12 2015.07.15D10:06:04.631217000
2 | 65 10.02 10.03 73 2015.07.15D10:06:04.631217000
3 | 51 4.98 4.99 225 2015.07.15D10:06:04.631217000
4 | 896 1.955 1.96 351 2015.07.15D10:06:04.631217000
q)trade
id| ts tp time
--| ----------------------------------------
1 | 7 80.15 2015.07.15D10:06:04.631217000
2 | 14 10.03 2015.07.15D10:06:04.631217000
0 | 8 110.15 2015.07.15D10:06:04.631217000
4 | 322 1.96 2015.07.15D10:06:04.631217000
3 | 41 4.98 2015.07.15D10:06:03.631217000
Functions
q).sim.path
{[s;r;t]
z:.stat.norminv count[t]?1f;
p:prds .stat.gbm[s;r;deltas[first t;t]] z;
p}
q).sim.genp
{[id;S;s;r;dtm]
t:abs type dtm;
tm:("np" t in 12 14 15h)$dtm;
p:S*path[s;r;tm%365D06];
c:`id,`time`date[t=14h],`price;
p:flip c!(id;dtm;p);
p}
q).sim.genq
{[ts;qs;p]
q:p,'flip `bp`ap!tickrnd[ts] p `price;
q:q,'flip `bs`as!1+count[p]?/:2#qs;
q:`id`time`bs`bp`ap`as#q;
q}
q).sim.gent
{[pct;q]
q:filter[pct] raze (-1_@[;`time;delay] q@) each group q `id;
t:q,' flip `ts`tp!trd[n?0b;(n:count q)?1f] . q `bs`bp`ap`as;
t:`id`time`ts`tp#t;
t}
Profiling
- Line Level - Scripts
- Function Level - CEP
- Instrumentation
Timing a Function
-
Create table to record name, start and end time
q)prof.events:flip `id`pid`func`time!"jjsn"$\:() q)prof.events id pid func time ---------------- -
Keep track of call id and parent call id
pid:id:0 - Record the start time
- Set call id and parent call id
- Call function (which may call other functions)
- Reset parent call id
- Record the end time and save the results
.prof.time:{[n;f;a]
s:.z.p;
id:.prof.id+:1;
pid:.prof.pid;
.prof.pid:id;
r:f . a;
.prof.pid:pid;
`prof.events upsert (id;pid;n;.z.p-s);
r}
q).prof.time[`foo;{x+y};1 2]
3
q)prof.events
id pid func time
--------------------------------
1 0 foo 0D00:00:00.000028000
Instrumenting a Function
- Obtain the **m**eta information from a **f**unction **n**ame
- Change to the function’s directory
- Redefine the function as a composition of
.prof.timeandenlist - Change back to root directory
.prof.instr:{[n]
m:get f:get n;
system "d .",string first m 3;
n set (')[.prof.time[n;f];enlist];
system "d .";
n}
q)f:{x+y}
q).prof.instr `f
`f
q)f
{[n;f;a]
s:.z.p;
id:.prof.id+:1;
pid:.prof.pid;
.prof.pid:id;..enlist
q)type f
105h
q)count get f
2
q)first get f
{[n;f;a]
s:.z.p;
id:.prof.id+:1;
pid:.prof.pid;
.prof.pid:id;
r:f . a;
.prof.pid:pid;
`prof.events upsert (id;pid;n;.z.p-s);
r}[`f;{x+y}]
q)last get f
enlist
q)enlist[1;2;3;4;5;6;7;8;9;10]
1 2 3 4 5 6 7 8 9 10
Profiling the CEP Engine
q).prof.instrall`
`..genu`..main`.util.use`.util.wday`.util.rng`.util.rnd`.util.ran..
q)prof.rpt
func | time n nc timepc pct
--------------| ----------------------------------
.timer.run | 9.751 118 1 0.08263559 34.30914
.md.updq | 3.714 40 1 0.09285 13.0678
.md.updp | 3.241 50 2 0.06482 11.40354
.md.updt | 2.73 28 1 0.0975 9.605573
.timer.until | 2.44 118 1 0.02067797 8.585201
.stat.tnorminv| 1.764 50 1 0.03528 6.206678
.stat.cnorminv| 1.576 50 2 0.03152 5.545195
.stat.norminv | 1.411 50 2 0.02822 4.964639
.stat.horner | 0.95 150 0 0.006333333 3.342599
.stat.gbm | 0.496 50 0 0.00992 1.745188
.sim.tickrnd | 0.201 40 0 0.005025 0.7072235
.sim.trd | 0.147 28 0 0.00525 0.5172232