path: root/src/NOTES.txt

I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html

The general structure of SledjHamr is a bunch of servers talking to each
other via Internet (or just local) connections.  One of them is a web
server for assets, world data, and inventory. 

Originally I didn't think using a web based world client was a good idea,
however it might be better to have one, for reasons.  Now I need a web
management console that can do all the things the current tmux console
can, including OpenSim console and commands.  Plus account management for
users.  I can also use a web based Jabber / XMPP front end to chat, IM,
and group chatter, which would run in the normal viewers web browser. 
This provides a doorway into putting SledjHamr stuff in existing viewers
without needing them to support it.  So a web based viewer now makes more
sense, and also means we can get away with not needing a viewer at all.

Toybox itself doesn't include a web server, and I don't think there is
one on the roadmap.  So we have to use an external web server, which was
a design goal of SledjHamr in the first place, using existing mature
HTTP infrastructure, coz that's already solved problems for a bunch of
things that plague OS/SL to this day.  Clear your cache!  Pffft.

So sledjchisl.c will be the "love world server", though initially it just
drives OpenSim_SC in tmux via tmux commands to send keys and read output. 
Later it might run opensim_SC directly and use STDIN and STDOUT to do
everything.  It'll also provide the text management front end that runs
in the left tmux panel of the first window, which is why it's based on
boxes in the first place.  Later still it can take over opensim_SC
functions as I move them out of mono.

We will need a text, web, and GUI version of this management front end. 
Hmmm, maybe don't need a GUI version, GUI users can just run a terminal.

After much research, FastCGI / FCGI seems to be the most portable way of
interfacing with existing web servers.  FCGI protocol closes STDERR and
STDOUT, and uses STDIN as it's two way communications channel to the web
server, so our FCGI module can't be used as the text management front
end.  This is probably a good idea to keep them separate anyway, for
security, coz the web server is exposed to the world, the console isn't.

Currently sledjchisl.c tests to see if it's running in tmux already, if
it isn't it starts up tmux runs itself into this new tmux, then exits. 
So it could also test if it's running from FCGI, and switch to web mode,
then it'll need to find the tmuxed instance to send commands to it. 
Either via nails connection, or sending tmux commands via shell.

FCGI has methods of dealing with auth and templates.  B-)

So for now I think I'll have the text and web management front ends in
sledjchisl.c, and the love world server as well.  I can split them up
later if I need to.




I has	Apache 2.4.25-3+deb9u9
	MariaDB 10.1.44-MariaDB


https://gist.github.com/dermesser/e2f9b66457ae19ebd116
    Multithreaded example in C.


-------------------------------------------------------------------

Apache doesn't seem to support FCGI filter role, so I might have to do
without.  Might be better anyway.


"A Filter is similar in functionality to a Responder that takes a data
file as a parameter. The difference is that with a Filter, both the data
file and the Filter itself can be access controlled using the Web
server's access control mechanisms, while a Responder that takes the name
of a data file as a parameter must perform its own access control checks
on the data file."

    Which is fine, our access control checks will be "Is this database
    defined user already logged on via our FCGI script?".  We should have
    total control over that.  I was planning on using the FCGI auth
    mechanism anyway.


RESPONDER
    web server sends FCGI_PARAMS
	CONTENT_LENGTH
    web server sends input body FCGI_STDIN
    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
	it has to finish reading FCGI_PARAMS first
    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)


FILTER
    filtered file has last modified time
	web server sets FCGI_DATA_LAST_MOD accordingly
    web server sends FCGI_PARAMS
	CONTENT_LENGTH   FCGI_DATA_LAST_MOD   FCGI_DATA_LENGTH
    web server sends input body FCGI_STDIN
    web servers sends file over FCGI_DATA
    fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD
    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
	it has to finish reading FCGI_STDIN first, but not FCGI_DATA
    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)


Soooo, FILTER might be slower anyway if we are caching the filtered file,
or mmapping it, coz filter has to start sending the filtered file, even
if it's to be served from cache.  Plus no need to wait for FCGI_STDIN
before spewing it out.


Last update time for parameters, plus an update frequency.  Once a minute.

    NOTE - SSI is a bit more complex than what I'm currently using.
	https://en.wikipedia.org/wiki/Server_Side_Includes
	    <!--#include virtual="menu.cgi" -->
	    <!--#include file="footer.html" -->
	    <!--#exec cgi="/cgi-bin/foo.cgi" -->
	    <!--#exec cmd="ls -l" -->
.	    <!--#echo var="REMOTE_ADDR" -->
	    <!--#config timefmt="%y %m %d" -->
	    <!--#config sizefmt="bytes" -->
	    <!--#config errmsg="SSI command failed!" -->
	    <!--#flastmod virtual="index.html" -->
	    <!--#fsize file="script.pl" -->
	    <!--#if expr="${Sec_Nav}" -->
	    <!--#include virtual="secondary_nav.txt" -->
	    <!--#elif expr="${Pri_Nav}" -->
	    <!--#include virtual="primary_nav.txt" -->
	    <!--#else -->
	    <!--#include virtual="article.txt" -->
	    <!--#endif -->
	    <!--#set var="foo" value="bar" -->
	    <!--#printenv -->
	https://www.w3.org/Jigsaw/Doc/User/SSI.html
	    Adds lots of others, including Java stuff.
	Mine
	    <!--#lua lua="print(table[key])" -->
	    <!--#lua file="/path/to/script.lua" -->
	    <!--#lua virtual="https://example.com/script.lua" -->

-------------------------------------------------------------------

Account creation process in the database.


UserAccounts table -
    UserFlags		64 is "allow gods to log in as me"
			0xf00 is membershipType, unless there's a title.  Only sent to viewers I think.
			32 is Minors   for estate banning purposes.
			4 is Anonymous for estate banning purposes.
			1 is AllowPublish  in profile, but userprofile has this as separate field.
			2 is MaturePublish in profile, but userprofile has this as separate field.
Presence table -
    UserID		varchar(255)
    RegionID		char(36)
    SessionID		char(36)
    SecureSessionID	char(36)
    LastSeen		timestamp
tokens table (I think this is actually used for something) -
    UUID		char(36)
    token		varchar(255)	current example looks like a UUID.
    validity		datetime
userdata (empty, can't find any actual usage in the source code, part of profiles) -
    UserId		char(36)	primary index
    TagId		varchar(64)	primary index
    DataKey		varchar(255)
    DataVal		varchar(255)
auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one.


    Vouched for
	userdata.UserId = UserAccounts.PrincipleID
	userdata.TagId = "vouches"
	userdata.DataKey = UUID of voucher
	userdata.DataVal = timestamp of vouching

	UserAccounts.Userlevel = -50
	UserAccounts.UserTitle = vouched for


-------------------------------------------------------------------

https://project-awesome.org/aleksandar-todorovic/awesome-c
    A curated list of C good stuff.

https://wolkykim.github.io/qdecoder/
    CGI library made by the qlibc guy, does support FCGI.
    Might be a wrapper around the fcgi_stdio stuff I'm already using?


https://danielmiessler.com/study/http/
    A Security-focused HTTP Primer
	Nothing much new except to say this about the Referer header -
	    "should not be used to make security decisions as it is controlled by the client"
	    Though others tell us to do precisely that.  lol

http://opensimulator.org/wiki/Userlevel

http://www.catb.org/esr/structure-packing/
    A good read, including a few links to other good stuff.
    Am I doing this stuff properly by intuition, or completely ballsing it up?  lol
    http://www.catb.org/esr/time-programming/

-------------------------------------------------------------------

apt install libmariadbclient-dev libapache2-mod-fcgid libssl1.0-dev uuid-dev spawn-fcgi

BTW, everything is some BSD licence variation, except MariaDB, which is
some combination of GPLv2, 2+, 3, and BSD variants.  With FOSS License
Exception, which seems to mean "Oracles GPL isn't viral".  Though my
license is "fuck you viral GPL, that's not your decision to make for my
fucking code" BSD.

-------------------------------------------------------------------

Install / update / upgrade.

I could keep the version number around.
    Include version numbers / branches of dependencies.
    Update will grab any security updates for the installed version.
    Upgrade   will upgrade   to a chosen later    different version.
    Downgrade will downgrade to a chosen earlier  different version.

Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the
main Luajit repo.  Everything else is on their master branches.

Bootstrap -
    bootstrap.sh or bootstrap.bat

    Build the LuaJIT that comes with our source.  It "builds out-of-the
    box on most systems" and has no dependencies, other than a C build system.

    Or download a prebuilt LuaJIT from somewhere.

  After toybox has been LuaJITized.

    Build the LuaJIT that comes with our source.  It "builds out-of-the
    box on most systems" and has no dependencies, other than a C build system.

    Similar should apply to toybox, though it's our LuaJITized version.
	Will need a specific miniconfig for this that doesn't include sledjchisl.

    Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo.

Install -
    install.lua

	Will need a pre flight check if the dependencies are installed.
	It checks if the system is built and has source.
	    Build it all.
	Do the usual copy stuff to a directory thing.
	Run "sledjchisl -install" in that directory.
	    Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running.
		The health check should include making sure our database creds exist / work.

Update / upgrade / downgrade
    install.lua -update
    install.lua -upgrade
    install.lua -downgrade

    Check if we are a binary only, or a source install.
	wget new binaries / git pull new source
	    Toybox has a wget in pending, otherwise it only has ftpget.
	    Git is standalone outside of the system, but if you are
	    running from source, you likely have standard build tools
	    like git.


Yeah I hate things that have their own packaging system, for needing to
step outside the operating systems packaging system, and adding to the too
long list of stuff I have to deal with manually, and now I are one.  lol


-------------------------------------------------------------------
-------------------------------------------------------------------

For logged in user, at the top show their name as linky to their accountView http://localhost/sledjchisl.fcgi/account.html?user=account+name
    That accountView offers edit / logout button, etc.
    Display account stuff, but not edit it until they hit the edit button.

When showing other users
    accountView, with edit / delete buttons if logged in user is high enough level.


-------------------------------------------------------------------

NEXT -

    Have the grid start up code also run the web backend inside the left over tmux panel.
	And have it restart if it crashes.

    Add the account.html stuff to the opensim-SC configuration, so viewers can get to it.
	ditto loginpage.html
	    retire all the OpenSim web stuff I added before
	write a stub page for the other pages viewers want

    Implement poorMansCron.html.

    Add the red asterisk to required fields, like every one else does.

    In var/cache/sessions, keep a uuid.lua full of the sessions for that user.
	Use it to clear out old sessions on login.
	Use it to clear out old sessions on validation.
	Use it to update the level in their session if some one changes their level.

-------------------------------------------------------------------

Should clean things up.
TODO -
    Move any side effects out of Validate functions, they should just stuff things into Rd->stuff.
	sessionValidate should be the only thing putting things into shs?
	    Nope, gotta get uuid, name, and level from database / uuid.lua when they log in.
    Move those side effects into Sub functions.

iF = accountPages->get(accountPages, form, NULL, false);	
..
sub = iF->subs->get(iF->subs, doit, NULL, false);		
..
  i = collectFields(Rd, iF, iV, t); 				Stuffs cookies, body, queries, NOT stuff, into iV (including source type).
// Validate the input data. 					Loops through iV.
    iV[i].field->validate(Rd, iF, &iV[i]);			Stuffs things into Rd->stuff.
	sessionValidate						Is special, ignores iV, gets things directly, reads the session.lua, stuffs things into shs and Rd->stuff.
	nameValidate						Also combines names into Rd->stuff, and into shs.
	passwordValidate					Also special...
	emailValidate						Stuffs both into stuff even if not validated.
..
// Submit the data.						TODO - do more stuff here, like login/out the account.  Login should check the password here and put things in shs.
    Usually -
	accountRead(Rd, iF, iV);
	complain if found / not found
	...
	freeSesh(Rd, FALSE, wipe);
	newSesh(Rd, FALSE);
    accountExploreValidatedVoucherSub				Does nothing.
..
// Return the result.						
    if no errors
	redirect to GET if it's POST, otcherwise -
	    find the output form
	    collect from stuff into iV
	call oF->web(Rd, iF, iV)
    else
	collect from stuff into iV
	call iF->eweb()

-------------------------------------------------------------------

Coffee Grid -

Destiny Grid -
    Auto add Hypergrid visitors group to "partner" grids.
	Estate has "Allow parcel access override?".  Not sure what that means.
	    Which does fuck all, and turns itself off.

Infinite Grid -
    Set up the deault member and their IAR.

-------------------------------------------------------------------
-------------------------------------------------------------------

BUGS!
-----

Check length in database values.

Names are case insensitive in world, should be on the web page to?
    I think they are on the database side, so I should store the Lua files with lower case file names, but use the case from within the files for display.
    I may have seen case insensitive grid logins fail, so should test this.
    Now I have seen them work. Viewer dependent?

The autogroup thing seems to have broke.  Doesn't work for gods.
    Or I did that on purpose, should check.  lol

Should limit viewing of other peoples account details, especially emails, to gods.

Clear out landmarks from the default IAR.

-------------------------------------------------------------------
-------------------------------------------------------------------

Hacks I should send up stream.

    qlibc/src/extensions/qconfig.c  line 402 - free(varstr)