From fccdf8c6f0f2a062ed35e93c5c322d781c3e6c55 Mon Sep 17 00:00:00 2001 From: onefang Date: Tue, 8 Sep 2020 21:38:19 +1000 Subject: SledjChisl notes. --- src/NOTES.txt | 383 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 383 insertions(+) create mode 100644 src/NOTES.txt diff --git a/src/NOTES.txt b/src/NOTES.txt new file mode 100644 index 0000000..a72e7c8 --- /dev/null +++ b/src/NOTES.txt @@ -0,0 +1,383 @@ +I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html + +The general structure of SledjHamr is a bunch of servers talking to each +other via Internet (or just local) connections. One of them is a web +server for assets, world data, and inventory. + +Originally I didn't think using a web based world client was a good idea, +however it might be better to have one, for reasons. Now I need a web +management console that can do all the things the current tmux console +can, including OpenSim console and commands. Plus account management for +users. I can also use a web based Jabber / XMPP front end to chat, IM, +and group chatter, which would run in the normal viewers web browser. +This provides a doorway into putting SledjHamr stuff in existing viewers +without needing them to support it. So a web based viewer now makes more +sense, and also means we can get away with not needing a viewer at all. + +Toybox itself doesn't include a web server, and I don't think there is +one on the roadmap. So we have to use an external web server, which was +a design goal of SledjHamr in the first place, using existing mature +HTTP infrastructure, coz that's already solved problems for a bunch of +things that plague OS/SL to this day. Clear your cache! Pffft. + +So sledjchisl.c will be the "love world server", though initially it just +drives OpenSim_SC in tmux via tmux commands to send keys and read output. +Later it might run opensim_SC directly and use STDIN and STDOUT to do +everything. It'll also provide the text management front end that runs +in the left tmux panel of the first window, which is why it's based on +boxes in the first place. Later still it can take over opensim_SC +functions as I move them out of mono. + +We will need a text, web, and GUI version of this management front end. +Hmmm, maybe don't need a GUI version, GUI users can just run a terminal. + +After much research, FastCGI / FCGI seems to be the most portable way of +interfacing with existing web servers. FCGI protocol closes STDERR and +STDOUT, and uses STDIN as it's two way communications channel to the web +server, so our FCGI module can't be used as the text management front +end. This is probably a good idea to keep them seperate anyway, for +security, coz the web server is exposed to the world, the console isn't. + +Currently sledjchisl.c tests to see if it's running in tmux already, if +it isn't it starts up tmux runs itself into this new tmux, then exits. +So it could also test if it's running from FCGI, and switch to web mode, +then it'll need to find the tmuxed instance to send commands to it. +Either via nails connection, or sending tmux commands via shell. + +FCGI has methods of dealing with auth and templates. B-) + +So for now I think I'll have the text and web management front ends in +sledjchisl.c, and the love world server as well. I can split them up +later if I need to. + + + + +I has Apache 2.4.25-3+deb9u9 + MariaDB 10.1.44-MariaDB + + +https://gist.github.com/dermesser/e2f9b66457ae19ebd116 + Multithreaded example in C. + + +------------------------------------------------------------------- + +Apache doesn't seem to support FCGI filter role, so I might have to do +without. Might be better anyway. + + +"A Filter is similar in functionality to a Responder that takes a data +file as a parameter. The difference is that with a Filter, both the data +file and the Filter itself can be access controlled using the Web +server's access control mechanisms, while a Responder that takes the name +of a data file as a parameter must perform its own access control checks +on the data file." + + Which is fine, our access control checks will be "Is this database + defined user already logged on via our FCGI script?". We should have + total control over that. I was planning on using the FCGI auth + mechanism anyway. + + +RESPONDER + web server sends FCGI_PARAMS + CONTENT_LENGTH + web server sends input body FCGI_STDIN + fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR + it has to finish reading FCGI_PARAMS first + fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) + + +FILTER + filtered file has last modified time + web server sets FCGI_DATA_LAST_MOD accordingly + web server sends FCGI_PARAMS + CONTENT_LENGTH FCGI_DATA_LAST_MOD FCGI_DATA_LENGTH + web server sends input body FCGI_STDIN + web servers sends file over FCGI_DATA + fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD + fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR + it has to finish reading FCGI_STDIN first, but not FCGI_DATA + fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) + + +Soooo, FILTER might be slower anyway if we are caching the filtered file, +or mmapping it, coz filter has to start sending the filtered file, even +if it's to be served from cache. Plus no need to wait for FCGI_STDIN +before spewing it out. + + +Last update time for parameters, plus an update frequency. Once a minute. + + NOTE - SSI is a bit more complex than what I'm currently using. + https://en.wikipedia.org/wiki/Server_Side_Includes + + + + +. + + + + + + + + + + + + + + + https://www.w3.org/Jigsaw/Doc/User/SSI.html + Adds lots of others, including Java stuff. + Mine + + + + +------------------------------------------------------------------- + +Account creation process in the database. + + +UserAccounts table - + UserFlags 64 is "allow gods to log in as me" + 0xf00 is membershipType, unles there's a title. Only sent to viewers I think. + 32 is Minors for estate banning purposes. + 4 is Anonymous for estate banning purposes. + 1 is AllowPublish in profile, but userprofile has this as separate field. + 2 is MaturePublish in profile, but userprofile has this as separate field. +Presence table - + UserID varchar(255) + RegionID char(36) + SessionID char(36) + SecureSessionID char(36) + LastSeen timestamp +tokens table (I think this is actually used for something) - + UUID char(36) + token varchar(255) current example looks like a UUID. + validity datetime +userdata (empty, can't find any actual usage in the source code, part of profiles) - + UserId char(36) primary index + TagId varchar(64) primary index + DataKey varchar(255) + DataVal varchar(255) +auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one. + + + Vouched for + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "vouches" + userdata.DataKey = UUID of voucher + userdata.DataVal = timestamp of vouching + + UserAccounts.Userlevel = -50 + UserAccounts.UserTitle = vouched for + + +------------------------------------------------------------------- + +https://project-awesome.org/aleksandar-todorovic/awesome-c + A curated list of C good stuff. + +https://wolkykim.github.io/qdecoder/ + CGI library made by the qlibc guy, does support FCGI. + Might be a wrapper around the fcgi_stdio stuff I'm already using? + + +https://danielmiessler.com/study/http/ + A Security-focused HTTP Primer + Nothing much new except to say this about the Referer header - + "should not be used to make security decisions as it is controlled by the client" + Though others tell us to do precisely that. lol + +http://opensimulator.org/wiki/Userlevel + +http://www.catb.org/esr/structure-packing/ + A good read, including a few links to other good stuff. + Am I doing this stuff properly by intuition, or completely ballsing it up? lol + http://www.catb.org/esr/time-programming/ + +------------------------------------------------------------------- + +apt install libmariadbclient-dev libapache2-mod-fcgid libssl1.0-dev uuid-dev spawn-fcgi + +BTW, everything is some BSD licence variation, except MariaDB, which is +some combination of GPLv2, 2+, 3, and BSD variants. With FOSS License +Exception, which seems to mean "Oracles GPL isn't viral". Though my +license is "fuck you viral GPL, that's not your decision to make for my +fucking code" BSD. + +------------------------------------------------------------------- + +Install / update / upgrade. + +I could keep the version number around. + Include version numbers / branches of dependencies. + Update will grab any security updates for the installed version. + Upgrade will upgrade to a chosen later different version. + Downgrade will downgrade to a chosen earlier different version. + +Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the +main Luajit repo. Everything else is on their master branches. + +Bootstrap - + bootstrap.sh or bootstrap.bat + + Build the LuaJIT that comes with our source. It "builds out-of-the + box on most systems" and has no dependencies, other than a C build system. + + Or download a prebuilt LuaJIT from somewhere. + + After toybox has been LuaJITized. + + Build the LuaJIT that comes with our source. It "builds out-of-the + box on most systems" and has no dependencies, other than a C build system. + + Similar should apply to toybox, though it's our LuaJITized version. + Will need a specific miniconfig for this that doesn't include sledjchisl. + + Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo. + +Install - + install.lua + + Will need a pre flight check if the dependencies are installed. + It checks if the system is built and has source. + Build it all. + Do the usual copy stuff to a directory thing. + Run "sledjchisl -install" in that directory. + Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running. + The health check should include making sure our database creds exist / work. + +Update / upgrade / downgrade + install.lua -update + install.lua -upgrade + install.lua -downgrade + + Check if we are a binary only, or a source install. + wget new binaries / git pull new source + Toybox has a wget in pending, otherwise it only has ftpget. + Git is standalone outside of the system, but if you are + running from source, you likely have standard build tools + like git. + + +Yeah I hate things that have their own packaging system, for needing to +step outside the operating systems packaging system, and adding to the too +long list of stuff I have to deal with manually, and now I are one. lol + + +------------------------------------------------------------------- +------------------------------------------------------------------- + +For logged in user, at the top show their name as linky to their accountView http://localhost/sledjchisl.fcgi/account.html?user=account+name + That accountView offers edit / logout button, etc. + Display account stuff, but not edit it until they hit the edit button. + +When showing other users + accountView, with edit / delete buttons if logged in user is high enough level. + + +------------------------------------------------------------------- + +NEXT - + + Have the grid start up code also run the web backend inside the left over tmux panel. + And have it restart if it crashes. + + Add the account.html stuff to the opensim-SC configuration, so viewers can get to it. + ditto loginpage.html + retire all the OpenSim web stuff I added before + write a stub page for the other pages viewers want + + Implement poorMansCron.html. + + Add the red asterisk to required fields, like every one else does. + + In var/cache/sessions, keep a uuid.lua full of the sessions for that user. + Use it to clear out old sessions on login. + Use it to clear out old sessions on validation. + Use it to update the level in their session if some one changes their level. + +------------------------------------------------------------------- + +Should clean things up. +TODO - + Move any side effects out of Validate functions, they should just stuff things into Rd->stuff. + sessionValidate should be the only thing putting things into shs? + Nope, gotta get uuid, name, and level from database / uuid.lua when they log in. + Move those side effects into Sub functions. + +iF = accountPages->get(accountPages, form, NULL, false); +.. +sub = iF->subs->get(iF->subs, doit, NULL, false); +.. + i = collectFields(Rd, iF, iV, t); Stuffs cookies, body, queries, NOT stuff, into iV (including source type). +// Validate the input data. Loops through iV. + iV[i].field->validate(Rd, iF, &iV[i]); Stuffs things into Rd->stuff. + sessionValidate Is special, ignores iV, gets things directly, reads the session.lua, stuffs things into shs and Rd->stuff. + nameValidate Also combines names into Rd->stuff, and into shs. + passwordValidate Also special... + emailValidate Stuffs both into stuff even if not validated. +.. +// Submit the data. TODO - do more stuff here, like login/out the account. Login should check the password here and put things in shs. + Usually - + accountRead(Rd, iF, iV); + complain if found / not found + ... + freeSesh(Rd, FALSE, wipe); + newSesh(Rd, FALSE); + accountExploreValidatedVoucherSub Does nothing. +.. +// Return the result. + if no errors + redirect to GET if it's POST, otcherwise - + find the output form + collect from stuff into iV + call oF->web(Rd, iF, iV) + else + collect from stuff into iV + call iF->eweb() + +------------------------------------------------------------------- + +Coffee Grid - + +Destiny Grid - + Auto add Hypergrid visitors group to "partner" grids. + Estate has "Allow parcel access overide?". Not sure what that means. + Which does fuck all, and turns itself off. + +Infinite Grid - + Set up the deault member and their IAR. + +------------------------------------------------------------------- +------------------------------------------------------------------- + +BUGS! +----- + +Check length in database values. + +Names are case insensitive in world, should be on the web page to? + I think they are on the database side, so I should store the Lua files with lower case file names, but use the case from within the files for display. + I may have seen case insensitive grid logins fail, so should test this. + Now I have seen them work. Viewer dependant? + +The autogroup thing seems to have broke. Doesn't work for gods. + Or I did that on purpose, should check. lol + +Should limit viewing of other peoples account details, especially emails, to gods. + +Clear out landmarks from the default IAR. + +------------------------------------------------------------------- +------------------------------------------------------------------- + +Hacks I should send up stream. + + qlibc/src/extensions/qconfig.c line 402 - free(varstr) -- cgit v1.1