Quest Scripting Language Features...

Chase
I know few like to hear my complain, so I try not to do it. However having used more grown up languages for so long I find quest's code features... or rather the lack there of... a little frustrating. Which is my only real qualm with the system (the rest I can more or less program around generally speaking).

So I thought adding a few minor features would make a programmers life a great deal less painful, I might even add them myself if I decide to get messy (and assuming said core is programmed in any sane way).

The short list (in order of importance)
[list]
[*]continue and break statements[/*:m]
[*]&& and || short circuiting[/*:m]
[*]+= -= /= *=[/*:m]
[*]++ -- prefix and postfix operators (I could live without these)[/*:m][/list:u]

I have written a few compilers, a few of which compiled to full blown bytecode rather then just IL. So I can provide some implementation details.

The Detailed List

break and continue statements

both break and continue work on the innermost loop structure
they work by pushing start/end loop targets to the stack,
if we encounter a break we pop both and use the end statement
if we encounter a continue we peek at the start statement


&& and || short circuiting

EXAMPLE
"if(a && b)" in this case, if a is false, b will not be tested and the if will be exited
"if(a || b)" in this case, if a is true, b will not be tested and the if will be exited


Stack Machine Bytecode of how this works.... (for an && statement)
-----------------------------------------
load_var a ;expression 1 code goes here
if_zero_goto label1
load_var b ;expression 2 code goes here
if_zero_goto label1
;; once we get here, both values are true
load_const 1
goto label2
_label1:
load_const 0
_label2:
;we load constants to the stack for further expression handling


Stack Machine Bytecode of how this works.... (for an || statement)
-----------------------------------------
load_var a ;expression 1 code goes here
if_not_zero_goto label1
load_var b ;expression 2 code goes here
if_not_zero_goto label1
;; once we get here, both values are true
load_const 0
goto label2
_label1:
load_const 1
_label2:
;we load constants to the stack for further expression handling


+= -= /= *=

We just expand these to slightly simpler expressions
+=
lvalue = lvalue + (expression)

-=
lvalue = lvalue - (expression)

/=
lvalue = lvalue / (expression)

*=
lvalue = lvalue * (expression)

Alex
All reasonable requests - I'm happy for you to implement them and send me a pull request if you like :)

I think implementing short-circuiting will require updating FLEE, which Quest uses to parse expressions. Quest uses my fork here: https://bitbucket.org/alexwarren/flee

Implementing things like += and ++ shouldn't be too difficult, but bear in mind they will need to be supported in the Script Editor too.

Chase
Alright i'll see what I can do in my free time.

I don't think I have heard of flee. Generally mostly use Antlr these days.

Anyway, one questions before I touch any expressions

How do you handle boolean operations?

Alex
Not sure I really understand the question, but Quest doesn't do anything special - every expression is handed to FLEE to parse, and Quest provides variable and function values when asked for them. So the answer is probably "whatever FLEE does".

Chase
EDITED
I missed a lot due to how tired I was. I have rewritten this a bit to fix my mistakes.

This concerns the addition of short-circuiting AND and OR.

Hah, well. Seems Flee is all vbscript. That syntax makes me head desk (I have only used vb for like 2 months back when vb6 was still about).

This is going to be fun.™

----

Okay.. at first glace it looks like Flee does support short circuiting out of the box, it just (apparently) isn't being used. I happen to know short circuiting doesn't work, or at least it didn't for me. But my tests of Quest Short Circuiting behavior were far from extensive.

----

I get an error of "Cannot convert type 'Int32' to expression result of 'Boolean'" when I test things in quest, which is interesting. I am guessing this is a problem in flee rather then Quest.

However under most circumstances flee uses the bitwise 'and' rather then the Boolean'and' in expressions. It does however sometimes seem to be using Boolean and when the two sides are Boolean.

The differences between Bitwise 'and' and Boolean 'and'.

c = 2 & 1
c is 0, since the bitwise and of 10b and 01b is 0

c = 2 && 1
c is 1, since neither value equals zero, which is what boolean (logical) and does


Let me see if I can force flee to just use the logical stuff. Since most people who write Quest code probably do not use the bitwise functionality anyway.

----

Flee design is interesting. But it could be better. Anyway as you can see in ExpressionElements.LogicalBitwise.AndOr.vb line 50 to 62, the method Emit. It does indeed use both logical and bitwise and.

I modified this method to only use only the logical versions and when set to numeric mode it does indeed seem to operate as expected (however missing variables still cause it to crash :/)

However there is a small bug in the AND routine, when evaluating something like "2 and 123" it will return 123, rather then 1. It is still technically a legal boolean true value, but I expect it is due to optimizations in the code, as flee only expects the Boolean type here. The OR works as expected however.

In conclusion
Flee has the capacity for logical 'and' and 'or'. We just have to somehow coax it to use it. So yes, it will likely require a modification of Flee to do.
The three options I see are as follows.

[list=1]
[*]We disable the ability for flee to use bitwise 'and' and bitwise 'or'. However this is a bit short sighted considering it has the capacity for it.[/*:m]
[*]We give bitwise 'and and bitwise 'or' their own tokens in flee and set 'and' and 'or' tokens to be used only for the logical functions. However this may involve a great deal of work (depending on how they did it).[/*:m]
[*]We completely replace flee with something else. While this has long term benefits, the short term requirement is considerable. I don't recommend doing unless you really think it worth the time. Creating just an expression parser is pretty easy though, so it might not be to bad.[/*:m][/list:o]

Either way I also think we need to enable Flee to be able to convert a numeric result to a boolean result. That is if the number is into zero it is true, if it is zero, then it is false.

Chase
Flee is not that well designed. Changing or adding anything requires making changes in a dozen places throughout the code. That, the strange customized version of Grammatica it requires, it being in Visual Basic convinced me it might be worth seeing if an alternative exists.

One does, better yet, it is almost a drop-in replacement. It is NCalc. It is made in Antlr, C# and has much cleaner code (just don't look in Numbers.cs but that can be fixed if we use .Net +4.0).

Just letting you know, it might be worth a glance. I know replacing a major component of any grown program is a time consuming problem, I plan to do a lot of the work and testing myself. In the free time I can spare anyway. :)

I wanted to get it working before I did any show and tell though. But I am mentioning it so if you object I don't spend to much of my non work days on it to find it out it's a no go.

Chase
Here is a sample of a heavily modified NCalc

http://www.mediafire.com/?pv1svxo2tf2fagv

I rewrote the grammar (since that is very easy with antlr) to make it more Quest friendly.

I added assignment code, since NCalc didn't have any. It wasn't too difficult.

It supports = += -= /= %= *= |= &= ^= <<= and >>= as far as assignment goes and it can do it inline.

Meaning this is a valid expression.

b = 4 - (a += 2)

I haven't hooked up the assignment to its parameter storage yet, since I was considering removing that entirely and just have a handler. so whoever is running it gets the request for the variable or told a variable has been assigned.

Other changes vs Flee

equality has changed from '=' to '=='
'^' has been reassigned from raising to a power to Bitwise Excusive Or (XOR)

'and' is no longer shared between bitwise and logical use
'&' is now used for Bitwise And

'or' is no longer shared between bitwise and logical use
'|' is now used for Bitwise Or

It has the capacity for short circuiting on Logical And and Logical Or

You gain '~' Bitwise Not as well as '!' for Logical Not ('not' is still valid for Logical Not)

Raising to a power is not implemented (but if you want it, it is easy enough to add).


There was an interesting problem, since I wanted to originally make it identical to current Flee syntax, however there was a flaw.
It cannot tell the difference between variable = 10 and variable = 10. To be fair, neither can I. So I don't really blame it much. However assignment and equality checking require separate tokens for this reason. Since it only handles expressions, it cannot guess from context.

Once I realize that would pretty much break all Quest code in existence, I decided a few more minor changes wouldn't hurt to badly. However could increase compatibility by changing ^ back to power and changing the bitwise xor to 'xor'.


Edit: I'll work more on this maybe Wednesday, see about hammering it into quest and updating all of the required parts of WorldModel to get it to work.

The Pixie
I like the sound of this. += etc. would be a great addition to Quest, and I would prefer == for equality as it would be the same as every other langiage I use, and I would feel more confident return (a == b) was returning a boolean, rather than b.

I would want to have some operator for raising to the power; it occasionally useful.

Whether others would consider the change in syntax I do not know. Would a translation program be possible? It would mean maintaining two versions of libraries. Would it be possible to update flee to use both the new and old syntax as a first step?

Chase
The Pixie wrote:I like the sound of this. += etc. would be a great addition to Quest, and I would prefer == for equality as it would be the same as every other langiage I use, and I would feel more confident return (a == b) was returning a boolean, rather than b.

I prefer '==' as well.

The Pixie wrote:I would want to have some operator for raising to the power; it occasionally useful.

What do you think of b = 2 pow 4 syntax wise?
The only other ways I see it done are b = 2 ** 4 and b = 2 ^^ 4, and of course b = 2 ^ 4.

In my humble opinion symbol based causes less confusion in the code, but ultimately not my decision. :)

The Pixie wrote:Would a translation program be possible? It would mean maintaining two versions of libraries.

A translation program would mean we wouldn't need to maintain two different libraries. The alternate choice would just to keep Flee around for older games. This would be the most compatible choice obviously, but adds a level of complexity. Luckily, we would never have to touch Flee again.

A translation program between the two formats wouldn't be inherently difficult. I will see about adding it to my queue of things to do. Since I am the one suggesting this, probably will be the one who ends up writing it.


Basic rules for updating (subject to tweaking of course):
change every = to == unless it is by itself on a line and the lvalue is an identifier
change every and outside of an if statement into an '&'
change every or outside of an if statement into an '|'
change every ^ to whatever syntax we choose
change every xor to ^


The Pixie wrote:Would it be possible to update flee to use both the new and old syntax as a first step?

Short answer. No.

Medium answer. It would probably be easier to update NCalc to guess at '=' and '='.

Long answer. I really don't want to. I already tried, in fact it was the first thing I tried. Before I even knew NCalc existed. Any change requires a dozen changes in other places, and the errors added up very quickly. That was just trying to separate the bitwise and logical 'and' and 'or' from each other. That is to say, Flee is a maintenance nightmare.

Alex
Just a few thoughts...

I would like to keep the syntax as similar as possible, otherwise there will be problems when people upgrade their games from an older version of Quest.

In any case I would like to keep FLEE for older game versions. We should use a factory to create Expression objects, then games with WorldModelVersion <= v530 (for example) could use FLEE and newer games can use the new expression evaluator. (This would be a change for v5.4 or later as v5.3 is big enough as it is)

We do some amount of pre-processing of expressions anyway, which enables us to handle things like spaces in object names. This means even if the expression evaluator requires double-equals, we can keep the Quest syntax using single-equals, and simply convert internally when the expression is loaded. I prefer single-equals for Quest as the language is designed to be easy for beginners, and it seems rare that one would want to use assignment in the middle of an expression.

Does NCalc give us control over object dot notation? That has always been a bit of a pain with FLEE and the current workaround is a bit cumbersome.

It would be worthwhile doing a performance comparison, as of course the expression evaluator is used very heavily.

Chase
Alex wrote:I would like to keep the syntax as similar as possible, otherwise there will be problems when people upgrade their games from an older version of Quest.


I would like a version that has a more advanced syntax, such as a library writer wants to be able to do b = 6 * ( a-= 4) or something. Not sure if a hidden toggle would be viable. Such as a scripting attribute on the asl tag. It would mean it would open the platform to more serious programmers who wouldn't want to be burdened by the restrictions of a simpler syntax.

Just means in you would do less script translation for them or their libraries.

(P.S. I am such a user.)

Alex wrote:In any case I would like to keep FLEE for older game versions. We should use a factory to create Expression objects, then games with WorldModelVersion <= v530 (for example) could use FLEE and newer games can use the new expression evaluator. (This would be a change for v5.4 or later as v5.3 is big enough as it is)


Well keeping FLEE for older games would be the most compatible.

Alex wrote:We do some amount of pre-processing of expressions anyway, which enables us to handle things like spaces in object names. This means even if the expression evaluator requires double-equals, we can keep the Quest syntax using single-equals, and simply convert internally when the expression is loaded. I prefer single-equals for Quest as the language is designed to be easy for beginners, and it seems rare that one would want to use assignment in the middle of an expression.


It is only rare for beginners. It can save a great deal of effort in some routines. See the first point.

As for spaces in identifiers. NCalc might be able to be modified to allow that natively, since it only handles the expressions. Since spaces are only whitespace, if it sees (identifier) (some amount of spaces) (identifier) it would know it is all one identifier. However identifiers starting or ending with spaces would be right out.

Have I mentioned modifying the grammar is easy?

Alex wrote:Does NCalc give us control over object dot notation? That has always been a bit of a pain with FLEE and the current workaround is a bit cumbersome.

NCalc has no idea what dot notation is. Adding it however is not a problem and actually easily done. It could even be done in the parser. Could have it directly ask Quest.. "What is 'player' 'parent' 'name'". or it could just pass the 'player.parent.name' variable request directly to quest.

I not sure I want to use the internal variable storage of NCalc, since it isn't very accessible, and could just pass all variable read and write requests to Quest to handle.

Alex wrote:It would be worthwhile doing a performance comparison, as of course the expression evaluator is used very heavily.


Well it depends if you currently cache the bytecode results from FLEE, or if FLEE does it automatically. NCalc has a built in cache system for post parsed expressions, but doesn't use IL, but rather tree evaluation. This would only likely effect repeat evaluation, since the slowest part of any expression evaluator is its parsing. Antlr isn't known for it's speed, but it is likely as fast as any other LL parser (Grammatica).

If IL is a required part of it (like for say, the Web Player scripting, which couldn't handle parsing trees very easily), adding it to NCalc is very easy, as it already tree walks.

Alex
Chase wrote:

"Alex"

I would like to keep the syntax as similar as possible, otherwise there will be problems when people upgrade their games from an older version of Quest.



I would like a version that has a more advanced syntax, such as a library writer wants to be able to do b = 6 * ( a-= 4) or something. Not sure if a hidden toggle would be viable. Such as a scripting attribute on the asl tag. It would mean it would open the platform to more serious programmers who wouldn't want to be burdened by the restrictions of a simpler syntax.



I suppose what I really mean is that the new syntax should add to, rather than change, the existing expression syntax. It will be confusing if people have to change "=" to "==" with a new version for example.

I can see the argument for adding an attribute to the asl tag, but I think this will cause confusion - there would have to be two different versions for each code sample that somebody posted to the forum, for example.


Alex wrote:We do some amount of pre-processing of expressions anyway, which enables us to handle things like spaces in object names. This means even if the expression evaluator requires double-equals, we can keep the Quest syntax using single-equals, and simply convert internally when the expression is loaded. I prefer single-equals for Quest as the language is designed to be easy for beginners, and it seems rare that one would want to use assignment in the middle of an expression.


It is only rare for beginners. It can save a great deal of effort in some routines. See the first point.



I can see that using -= and += in an expression can be useful, and this could be added while retaining full compatibility with existing expressions. So I'd be fine with that.

I don't see the point of using single-equals for assignment within an expression though, so I think we can keep using single-equals for equality.


As for spaces in identifiers. NCalc might be able to be modified to allow that natively, since it only handles the expressions. Since spaces are only whitespace, if it sees (identifier) (some amount of spaces) (identifier) it would know it is all one identifier. However identifiers starting or ending with spaces would be right out.



Sounds good. The only thing to watch out for are operators such as "and" and "or". Quest joins words together as part of a single identifier, except where one of the words is one of these keywords. Not sure how easy it would be for NCalc to take that into account. It also means we have to be careful about adding new operators - although I think something like "pow" is unlikely to cause problems. But we should prefer symbols as far as possible.


Alex wrote:Does NCalc give us control over object dot notation? That has always been a bit of a pain with FLEE and the current workaround is a bit cumbersome.

NCalc has no idea what dot notation is. Adding it however is not a problem and actually easily done. It could even be done in the parser. Could have it directly ask Quest.. "What is 'player' 'parent' 'name'". or it could just pass the 'player.parent.name' variable request directly to quest.



Excellent - there's currently a workaround to make this work with FLEE, so anything that makes the code simpler (at least for newer games) is great.


Alex wrote:It would be worthwhile doing a performance comparison, as of course the expression evaluator is used very heavily.


Well it depends if you currently cache the bytecode results from FLEE, or if FLEE does it automatically. NCalc has a built in cache system for post parsed expressions, but doesn't use IL, but rather tree evaluation. This would only likely effect repeat evaluation, since the slowest part of any expression evaluator is its parsing. Antlr isn't known for it's speed, but it is likely as fast as any other LL parser (Grammatica).

If IL is a required part of it (like for say, the Web Player scripting, which couldn't handle parsing trees very easily), adding it to NCalc is very easy, as it already tree walks.



We compile with FLEE the first time an expression is evaluated, and recompile if any of the types in the expression have changed. This applies to both desktop and WebPlayer as the back-end is exactly the same (web-based games run on the server - only UI stuff happens on the client).

I guess we will only really know once there is an NCalc version of the code, then some performance profiling can be done.

Chase
Alex wrote:
I suppose what I really mean is that the new syntax should add to, rather than change, the existing expression syntax. It will be confusing if people have to change "=" to "==" with a new version for example. I can see the argument for adding an attribute to the asl tag, but I think this will cause confusion - there would have to be two different versions for each code sample that somebody posted to the forum, for example.

I can see that using -= and += in an expression can be useful, and this could be added while retaining full compatibility with existing expressions. So I'd be fine with that. I don't see the point of using single-equals for assignment within an expression though, so I think we can keep using single-equals for equality.


Okay okay, I relent. I guess I can deal with = double duty. Which I can think of a few cases where just '=' inside can be used. I guess having it out makes the code easier to read.

Alex wrote:Sounds good. The only thing to watch out for are operators such as "and" and "or". Quest joins words together as part of a single identifier, except where one of the words is one of these keywords. Not sure how easy it would be for NCalc to take that into account. It also means we have to be careful about adding new operators - although I think something like "pow" is unlikely to cause problems. But we should prefer symbols as far as possible.


And and Or are only a problem if they are by themselves, surrounded by spaces. "man named andy" would by okay, as would 'more powerful axe', even 'this andor that' would be okey-dokey by the parser. In fact, even "this and_or that". What would not be okay would be something like "he or she" or "them and me". Get what I mean?

Alternatively, we could just surround the entire variable in some kind of bracket before sending it to the expression parser, so that "them and me" becomes "[them and me]" and I can configure the parser not to give a hoot. Means you could even have "1 % 2 and 3 + 78 = 43" as a variable name and it would be okay with that (though I have no idea how you would work that into the asl side).

Alex wrote:
We compile with FLEE the first time an expression is evaluated, and recompile if any of the types in the expression have changed. This applies to both desktop and WebPlayer as the back-end is exactly the same (web-based games run on the server - only UI stuff happens on the client).


In theory, with in NCalc you would not have to recompile the tree if the type changes. If I configure the IL right, not even if we go the IL route.

Alex wrote:I guess we will only really know once there is an NCalc version of the code, then some performance profiling can be done.


Yeah sorry, gotta eat, so gotta work (for money). I'll see what I can hammer out tomorrow.

Chase
I decided to go with the [ and ] method, and reserving normal identifiers only for functions. This simplifies problems with spaces and even starting or ending variables with spaces.

Current Prototype Output

[a] = 4 => 4
2 * (3 + 5) => 16
2 * (2*(2*(2+1))) => 24
10 % 3 => 1
false or not (false and true) => True
3 > 2 and 1 <= (3-2) => True
3 % 2 != 10 % 3 => 1
3 and 4 => True
[age] => 14
[ no really and what be or age? ] => 24
[age] >= 18 => False
if( [age] >= 18, 'majeur', 'mineur') => mineur
if( [age]+6 >= 18, 'majeur', 'mineur') => majeur


I went with the [ and ] method. Since it was something that NCalc had originally anyway and it makes sense here. It also simplifies the identifier lexer code by a great deal. The only illegal character within a variable name is now ']', which I am sure we could come up with a simple way to fix if needed. This would need to be retweaked if you wanted to add say [expression] based arrays.

I do not honestly expect Quest to be able to support the full range of characters NClac variables can have. Since quest code does not have [ and ] delimiting, and uses dot notation. But [_@#$a-zA-Z0-9]([ ]*[_@#$a-zA-Z0-9]+)* is realistic.

Function names follow the old rules, that is to say function names are limited to being [_a-zA-Z][_a-zA-Z0-9]*

Currently the only bug currently is:
3 % 2 != 10 % 3 => 1

Which I need to look into.


Current built in functions are

Abs
Acos
Asin
Atan
Ceiling
Cos
Exp
Floor
IEEERemainder
Log
Log10
Pow
Round
Sign
Sin
Sqrt
Tan
Truncate
Max
Min
if
in


These can be however easily renamed, removed, added to or altered.

Chase
I completely forgot Quest syntax, so that != wasn't working because Quest doesn't have a symbol for !=. Not Equals is only <>.


3 % 2 => 1
10 % 3 => 1
3 % 2 == 10 % 3 => True
3 % 2 <> 10 % 3 => False


It didn't complain at all, so I am wondering it that is a good thing or a bad thing.

I am not sure how it handles it currently, so reverting it might require some diving into exactly why it ignores the errors. I know in the Java version of Antlr parsers and lexers, errors are thrown, so it either C# or something NCalc does.

Chase
Well I got it to compile with NCalc, however that means nothing what so ever, since I don't correctly handle Expression<T> yet. Since I don't use C# I am not sure how to dynamically convert obj to T if obj is not already T.

I honestly have no idea what all the type gobbly gook is for, so I likely gummed things up there.

Inside Expression<T>

public T Execute(Context c)
{
NCalc.Expression expr = new NCalc.Expression(m_expression);
expr.EvaluateFunction += m_scriptContext.EvaluateFunctionHandler;
expr.EvaluateVariable += m_scriptContext.EvaluateVariableHandler;

object obj = expr.Evaluate();

//this will likely not work, but lets do it anyway
return (T)obj;
}


Inside ExpressionGeneric

public object Execute(Context c)
{
NCalc.Expression expr = new NCalc.Expression(m_expression);
expr.EvaluateFunction += m_scriptContext.EvaluateFunctionHandler;
expr.EvaluateVariable += m_scriptContext.EvaluateVariableHandler;
//seems sane enough
return expr.Evaluate();
}


Inside ScriptContext

/**
* Connector for NCalc function calls.
*/
private void FunctionHandler(string name, FunctionArgs args)
{
Element proc = m_worldModel.Procedure(name);
Parameters parameters = new Parameters();

int cnt = 0;

// TO DO: Check number of parameters matches
foreach (object val in args.EvaluateParameters())
parameters.Add((string)proc.Fields[FieldDefinitions.ParamNames][cnt++], val);

args.Result = m_worldModel.RunProcedure(name, parameters, true);
}

/**
* Connector for NCalc variable calls.
*/
private void VariableHandler(string name, VariableArgs args)
{
if (args.IsAssignment)
{
//This will probably not work, but not sure how quest handles this yet
m_context.Parameters[name] = args.Result;
}
else
{
args.Result = ResolveVariable(name);
}
}



If you want to try it yourself, here is the newest version of my (heavily) modified NCalc.

DOWNLOAD NCALC + SRC


I am actually not sure how to go about testing to be honest. Not to mention modify it to do the required = to == conversion.

Chase
Seems the [ and ] method isn't going to pan out, since it would require some work on the Quest side of things to get it to work. Right now I have it working with variables without spaces. But variables with spaces are being annoying.

In theory I could fix this by changing the identifier rule from a lexer to a parser rule. So until I have that working I don't want to hack on Quest to much more.


Current rule definition is:

fragment IdentFragment
: Letter (Letter | Digit)*
;

Ident
: IdentFragment+ ('.' IdentFragment+)*
;

Chase
After watching http://www.youtube.com/watch?v=taaEzHI9xyY which I recommend watching it if you are a programmer.

I have decided to stop working on this as long as = is == and =. While this video is mostly about how a programmer can do better, I think we can allow the programmer to do better by making the language better.

Forms that can hide defects are considered defective.

Alex
What do you mean? Which bit of that video is relevant here...?

Chase
Did you watch it?

Well aside, = having two different uses based on context is more confusing then just having = and == having their own uses.

Most of the entire reason I started working on NCalc was to get rid of that issue.

Since you want to continue using flee anyway for older code, I don't see any real benefit to keeping the older style. You can still use what I have, and I will even post up the quest code modifications I have done.

But I don't really want to continue working on something that doesn't really do what I wanted it to do.

This topic is now closed. Topics are closed after 60 days of inactivity.

Support

Forums