deal with symbols that don't appear in an equation

[albop]

This is a follow up to EconForge/Dolo.jl#26

Suppose we have a vector of symbols like states=[:a,:b,:c] and the the equation a+b +c + c(1). Imagine, we have an equation type whose signature is: g(states(-1), states, states(1))

By default (in dolo.jl and dolo.py), we would compile a function g(x, y, z) where x,y,z are all vectors of the same size (of length 3). But this could lead to suboptimal computation since:

• no variable appear with date t-1
• some variables don't appear with date t+1 (only c needs to be known)

I see two way of adressing the problem:

• change dolo.py and dolo.jl, so that another version of g is computed which involve only the variables that are actually used. Concretely, x would be expectd to be of size 0, y of size 3 and z of size 1. This fixes all drawbacks, but algorithms would seem a bit less natural. It would also probably require several versions of the same function.
• introduce dummy arguments for the variables that are not going to be used:
  1/ in the case of a whole vector being ommited we could allow g(nothing, y,z) (resp g(None,y,z)) to work as expected.
  2/ introduce a new class of vectors with non contiguous indexing (for use in our functions). For instance, zz could be stored in memory as (v_a), but zz[3] would interpreted in the generated function as zz[1].

I don't know whether this is completely crazy. It would seem natural to me to write, g(nothing, y, z) or g(x,y,Packed(zz,(3,))) if zz is a one element vector. The Packed object would only be intended to change the behaviour in g().

@spencerlyon2 : any thought ?

[sglyon]

While I'm all for making things more efficient, I don't quite see how the proposed change is usable in practice.

How would algorithm writers know which variables appear in which equations? Would we expect them to rely on our incidence tables and then only pack up variables that appear in the tables?

Another potential issue I see is different equations requiring a different subset of variables at a particular time. For example, in this particular equation only c appears at time t+1, but what if another equation had both a(1) and c(1)? Would the algorithm writer then need to package up two versions of the state at t+1 vector in order to call the two functions?

My last question is what type of efficiency gains we are hoping to achieve by doing this? Is it the memory requirement of keeping unused variables in vectors (or higher order arrays)?

Sorry if this sounds overly critical -- I don't think I have quite as clear of a vision for how this would be used.

[albop]

I don't mind the criticism, especially since these are not really issues ;-)

Yes, of course, the algorithm writer will have access to the incidence tables, both for each equation and for each group of equation. This is necessary anyway if you want to avoid the construction interpolators for controls that don't even appear in t+1. Efficiency gains can be quite big for models that have plenty of controls, some of them only being forward looking.

This is feasible by recompiling a special function or keeping the existing ones, by passing arrays with columns that will never be accessed.

Also, if you compile several equations, the incidence for this group of equations is the union of the individual incidences. In the above example if the two equations where a+b +c + c(1) and a+b +c + b(1) the vector states(1) would contain b(1) and c(1).

To be frank I don't know very well the right way to do it in Python. But on the julia side, it sounds easy. I guess, you just need specialize the function _unpack_expr so that it works differently on nothing and on packed arrays.