SledjChisl notes.

1 files changed, 383 insertions, 0 deletions

diff --git a/src/NOTES.txt b/src/NOTES.txt
new file mode 100644
index 0000000..a72e7c8
--- /dev/null
+++ b/src/NOTES.txt
@@ -0,0 +1,383 @@
+I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html
+
+The general structure of SledjHamr is a bunch of servers talking to each
+other via Internet (or just local) connections.  One of them is a web
+server for assets, world data, and inventory. 
+
+Originally I didn't think using a web based world client was a good idea,
+however it might be better to have one, for reasons.  Now I need a web
+management console that can do all the things the current tmux console
+can, including OpenSim console and commands.  Plus account management for
+users.  I can also use a web based Jabber / XMPP front end to chat, IM,
+and group chatter, which would run in the normal viewers web browser. 
+This provides a doorway into putting SledjHamr stuff in existing viewers
+without needing them to support it.  So a web based viewer now makes more
+sense, and also means we can get away with not needing a viewer at all.
+
+Toybox itself doesn't include a web server, and I don't think there is
+one on the roadmap.  So we have to use an external web server, which was
+a design goal of SledjHamr in the first place, using existing mature
+HTTP infrastructure, coz that's already solved problems for a bunch of
+things that plague OS/SL to this day.  Clear your cache!  Pffft.
+
+So sledjchisl.c will be the "love world server", though initially it just
+drives OpenSim_SC in tmux via tmux commands to send keys and read output. 
+Later it might run opensim_SC directly and use STDIN and STDOUT to do
+everything.  It'll also provide the text management front end that runs
+in the left tmux panel of the first window, which is why it's based on
+boxes in the first place.  Later still it can take over opensim_SC
+functions as I move them out of mono.
+
+We will need a text, web, and GUI version of this management front end. 
+Hmmm, maybe don't need a GUI version, GUI users can just run a terminal.
+
+After much research, FastCGI / FCGI seems to be the most portable way of
+interfacing with existing web servers.  FCGI protocol closes STDERR and
+STDOUT, and uses STDIN as it's two way communications channel to the web
+server, so our FCGI module can't be used as the text management front
+end.  This is probably a good idea to keep them seperate anyway, for
+security, coz the web server is exposed to the world, the console isn't.
+
+Currently sledjchisl.c tests to see if it's running in tmux already, if
+it isn't it starts up tmux runs itself into this new tmux, then exits. 
+So it could also test if it's running from FCGI, and switch to web mode,
+then it'll need to find the tmuxed instance to send commands to it. 
+Either via nails connection, or sending tmux commands via shell.
+
+FCGI has methods of dealing with auth and templates.  B-)
+
+So for now I think I'll have the text and web management front ends in
+sledjchisl.c, and the love world server as well.  I can split them up
+later if I need to.
+
+
+
+
+I has   Apache 2.4.25-3+deb9u9
+        MariaDB 10.1.44-MariaDB
+
+
+https://gist.github.com/dermesser/e2f9b66457ae19ebd116
+    Multithreaded example in C.
+
+
+-------------------------------------------------------------------
+
+Apache doesn't seem to support FCGI filter role, so I might have to do
+without.  Might be better anyway.
+
+
+"A Filter is similar in functionality to a Responder that takes a data
+file as a parameter. The difference is that with a Filter, both the data
+file and the Filter itself can be access controlled using the Web
+server's access control mechanisms, while a Responder that takes the name
+of a data file as a parameter must perform its own access control checks
+on the data file."
+
+    Which is fine, our access control checks will be "Is this database
+    defined user already logged on via our FCGI script?".  We should have
+    total control over that.  I was planning on using the FCGI auth
+    mechanism anyway.
+
+
+RESPONDER
+    web server sends FCGI_PARAMS
+        CONTENT_LENGTH
+    web server sends input body FCGI_STDIN
+    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
+        it has to finish reading FCGI_PARAMS first
+    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)
+
+
+FILTER
+    filtered file has last modified time
+        web server sets FCGI_DATA_LAST_MOD accordingly
+    web server sends FCGI_PARAMS
+        CONTENT_LENGTH   FCGI_DATA_LAST_MOD   FCGI_DATA_LENGTH
+    web server sends input body FCGI_STDIN
+    web servers sends file over FCGI_DATA
+    fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD
+    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
+        it has to finish reading FCGI_STDIN first, but not FCGI_DATA
+    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)
+
+
+Soooo, FILTER might be slower anyway if we are caching the filtered file,
+or mmapping it, coz filter has to start sending the filtered file, even
+if it's to be served from cache.  Plus no need to wait for FCGI_STDIN
+before spewing it out.
+
+
+Last update time for parameters, plus an update frequency.  Once a minute.
+
+    NOTE - SSI is a bit more complex than what I'm currently using.
+        https://en.wikipedia.org/wiki/Server_Side_Includes
+            <!--#include virtual="menu.cgi" -->
+            <!--#include file="footer.html" -->
+            <!--#exec cgi="/cgi-bin/foo.cgi" -->
+            <!--#exec cmd="ls -l" -->
+.           <!--#echo var="REMOTE_ADDR" -->
+            <!--#config timefmt="%y %m %d" -->
+            <!--#config sizefmt="bytes" -->
+            <!--#config errmsg="SSI command failed!" -->
+            <!--#flastmod virtual="index.html" -->
+            <!--#fsize file="script.pl" -->
+            <!--#if expr="${Sec_Nav}" -->
+            <!--#include virtual="secondary_nav.txt" -->
+            <!--#elif expr="${Pri_Nav}" -->
+            <!--#include virtual="primary_nav.txt" -->
+            <!--#else -->
+            <!--#include virtual="article.txt" -->
+            <!--#endif -->
+            <!--#set var="foo" value="bar" -->
+            <!--#printenv -->
+        https://www.w3.org/Jigsaw/Doc/User/SSI.html
+            Adds lots of others, including Java stuff.
+        Mine
+            <!--#lua lua="print(table[key])" -->
+            <!--#lua file="/path/to/script.lua" -->
+            <!--#lua virtual="https://example.com/script.lua" -->
+
+-------------------------------------------------------------------
+
+Account creation process in the database.
+
+
+UserAccounts table -
+    UserFlags           64 is "allow gods to log in as me"
+                        0xf00 is membershipType, unles there's a title.  Only sent to viewers I think.
+                        32 is Minors   for estate banning purposes.
+                        4 is Anonymous for estate banning purposes.
+                        1 is AllowPublish  in profile, but userprofile has this as separate field.
+                        2 is MaturePublish in profile, but userprofile has this as separate field.
+Presence table -
+    UserID              varchar(255)
+    RegionID            char(36)
+    SessionID           char(36)
+    SecureSessionID     char(36)
+    LastSeen            timestamp
+tokens table (I think this is actually used for something) -
+    UUID                char(36)
+    token               varchar(255)    current example looks like a UUID.
+    validity            datetime
+userdata (empty, can't find any actual usage in the source code, part of profiles) -
+    UserId              char(36)        primary index
+    TagId               varchar(64)     primary index
+    DataKey             varchar(255)
+    DataVal             varchar(255)
+auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one.
+
+
+    Vouched for
+        userdata.UserId = UserAccounts.PrincipleID
+        userdata.TagId = "vouches"
+        userdata.DataKey = UUID of voucher
+        userdata.DataVal = timestamp of vouching
+
+        UserAccounts.Userlevel = -50
+        UserAccounts.UserTitle = vouched for
+
+
+-------------------------------------------------------------------
+
+https://project-awesome.org/aleksandar-todorovic/awesome-c
+    A curated list of C good stuff.
+
+https://wolkykim.github.io/qdecoder/
+    CGI library made by the qlibc guy, does support FCGI.
+    Might be a wrapper around the fcgi_stdio stuff I'm already using?
+
+
+https://danielmiessler.com/study/http/
+    A Security-focused HTTP Primer
+        Nothing much new except to say this about the Referer header -
+            "should not be used to make security decisions as it is controlled by the client"
+            Though others tell us to do precisely that.  lol
+
+http://opensimulator.org/wiki/Userlevel
+
+http://www.catb.org/esr/structure-packing/
+    A good read, including a few links to other good stuff.
+    Am I doing this stuff properly by intuition, or completely ballsing it up?  lol
+    http://www.catb.org/esr/time-programming/
+
+-------------------------------------------------------------------
+
+apt install libmariadbclient-dev libapache2-mod-fcgid libssl1.0-dev uuid-dev spawn-fcgi
+
+BTW, everything is some BSD licence variation, except MariaDB, which is
+some combination of GPLv2, 2+, 3, and BSD variants.  With FOSS License
+Exception, which seems to mean "Oracles GPL isn't viral".  Though my
+license is "fuck you viral GPL, that's not your decision to make for my
+fucking code" BSD.
+
+-------------------------------------------------------------------
+
+Install / update / upgrade.
+
+I could keep the version number around.
+    Include version numbers / branches of dependencies.
+    Update will grab any security updates for the installed version.
+    Upgrade   will upgrade   to a chosen later    different version.
+    Downgrade will downgrade to a chosen earlier  different version.
+
+Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the
+main Luajit repo.  Everything else is on their master branches.
+
+Bootstrap -
+    bootstrap.sh or bootstrap.bat
+
+    Build the LuaJIT that comes with our source.  It "builds out-of-the
+    box on most systems" and has no dependencies, other than a C build system.
+
+    Or download a prebuilt LuaJIT from somewhere.
+
+  After toybox has been LuaJITized.
+
+    Build the LuaJIT that comes with our source.  It "builds out-of-the
+    box on most systems" and has no dependencies, other than a C build system.
+
+    Similar should apply to toybox, though it's our LuaJITized version.
+        Will need a specific miniconfig for this that doesn't include sledjchisl.
+
+    Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo.
+
+Install -
+    install.lua
+
+        Will need a pre flight check if the dependencies are installed.
+        It checks if the system is built and has source.
+            Build it all.
+        Do the usual copy stuff to a directory thing.
+        Run "sledjchisl -install" in that directory.
+            Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running.
+                The health check should include making sure our database creds exist / work.
+
+Update / upgrade / downgrade
+    install.lua -update
+    install.lua -upgrade
+    install.lua -downgrade
+
+    Check if we are a binary only, or a source install.
+        wget new binaries / git pull new source
+            Toybox has a wget in pending, otherwise it only has ftpget.
+            Git is standalone outside of the system, but if you are
+            running from source, you likely have standard build tools
+            like git.
+
+
+Yeah I hate things that have their own packaging system, for needing to
+step outside the operating systems packaging system, and adding to the too
+long list of stuff I have to deal with manually, and now I are one.  lol
+
+
+-------------------------------------------------------------------
+-------------------------------------------------------------------
+
+For logged in user, at the top show their name as linky to their accountView http://localhost/sledjchisl.fcgi/account.html?user=account+name
+    That accountView offers edit / logout button, etc.
+    Display account stuff, but not edit it until they hit the edit button.
+
+When showing other users
+    accountView, with edit / delete buttons if logged in user is high enough level.
+
+
+-------------------------------------------------------------------
+
+NEXT -
+
+    Have the grid start up code also run the web backend inside the left over tmux panel.
+        And have it restart if it crashes.
+
+    Add the account.html stuff to the opensim-SC configuration, so viewers can get to it.
+        ditto loginpage.html
+            retire all the OpenSim web stuff I added before
+        write a stub page for the other pages viewers want
+
+    Implement poorMansCron.html.
+
+    Add the red asterisk to required fields, like every one else does.
+
+    In var/cache/sessions, keep a uuid.lua full of the sessions for that user.
+        Use it to clear out old sessions on login.
+        Use it to clear out old sessions on validation.
+        Use it to update the level in their session if some one changes their level.
+
+-------------------------------------------------------------------
+
+Should clean things up.
+TODO -
+    Move any side effects out of Validate functions, they should just stuff things into Rd->stuff.
+        sessionValidate should be the only thing putting things into shs?
+            Nope, gotta get uuid, name, and level from database / uuid.lua when they log in.
+    Move those side effects into Sub functions.
+
+iF = accountPages->get(accountPages, form, NULL, false);        
+..
+sub = iF->subs->get(iF->subs, doit, NULL, false);               
+..
+  i = collectFields(Rd, iF, iV, t);                             Stuffs cookies, body, queries, NOT stuff, into iV (including source type).
+// Validate the input data.                                     Loops through iV.
+    iV[i].field->validate(Rd, iF, &iV[i]);                      Stuffs things into Rd->stuff.
+        sessionValidate                                         Is special, ignores iV, gets things directly, reads the session.lua, stuffs things into shs and Rd->stuff.
+        nameValidate                                            Also combines names into Rd->stuff, and into shs.
+        passwordValidate                                        Also special...
+        emailValidate                                           Stuffs both into stuff even if not validated.
+..
+// Submit the data.                                             TODO - do more stuff here, like login/out the account.  Login should check the password here and put things in shs.
+    Usually -
+        accountRead(Rd, iF, iV);
+        complain if found / not found
+        ...
+        freeSesh(Rd, FALSE, wipe);
+        newSesh(Rd, FALSE);
+    accountExploreValidatedVoucherSub                           Does nothing.
+..
+// Return the result.                                           
+    if no errors
+        redirect to GET if it's POST, otcherwise -
+            find the output form
+            collect from stuff into iV
+        call oF->web(Rd, iF, iV)
+    else
+        collect from stuff into iV
+        call iF->eweb()
+
+-------------------------------------------------------------------
+
+Coffee Grid -
+
+Destiny Grid -
+    Auto add Hypergrid visitors group to "partner" grids.
+        Estate has "Allow parcel access overide?".  Not sure what that means.
+            Which does fuck all, and turns itself off.
+
+Infinite Grid -
+    Set up the deault member and their IAR.
+
+-------------------------------------------------------------------
+-------------------------------------------------------------------
+
+BUGS!
+-----
+
+Check length in database values.
+
+Names are case insensitive in world, should be on the web page to?
+    I think they are on the database side, so I should store the Lua files with lower case file names, but use the case from within the files for display.
+    I may have seen case insensitive grid logins fail, so should test this.
+    Now I have seen them work. Viewer dependant?
+
+The autogroup thing seems to have broke.  Doesn't work for gods.
+    Or I did that on purpose, should check.  lol
+
+Should limit viewing of other peoples account details, especially emails, to gods.
+
+Clear out landmarks from the default IAR.
+
+-------------------------------------------------------------------
+-------------------------------------------------------------------
+
+Hacks I should send up stream.
+
+    qlibc/src/extensions/qconfig.c  line 402 - free(varstr)