Spezial:Permanentlink/64368286 sandbox for new created status/subster bot. This bot watches text on external (or internal) pages, the text is specified by regex. If this text changes, the bot copies the new/actual text into the wiki. Tag model to mark places of text drop is borrowed from MerlBot (and this sub-bot could also be used on my local wiki with an anacron-job ;).
Experimentally introduced a new bot part 'page_disc.py' for Spezial:Permanentlink/77229254#Disk-Zusammenfassung_auf_WD:ADT, test is running on Benutzer:DrTrigon/Spielwiese#Test 3. For this several small changes had to be made on 'dtbext_query.py' and 'dtbext_wikipedia.py' which should also return some benefit to other bot parts, e.g. 'sum_disc.py' (parser and others - look at the comments in the mentioned files).
MailerBot; the bugfixes in 'dtbext_query.py' enabled to send longer mails and thus it looks like the earlier introduced long mail support becomes obsolete. May be for very special needs it can still be helpful, but at the moment it is disabled.
MailerBot; enabled the bot to create and sent long mail through the toolserver for users, that have registered themselves.
mailer.py, mailerreg.py
Feature for:
SubsterBot; enabled the bot to retrieve wiki text content, not only html page content. Used for wiki to wiki text substitutions.
subster_beta.py, substersim.py
Bugfix for:
Small bug in MailerBot; the 'content_maxlen' has to include the subject size and thus must be calculated for every page (else some mails for certain pages cannot be sent).
mailer.py
Bugfix for:
Small bug in 'Pages.getVersionHistory(...)'; if the API omitts some entries (and sometimes it does) and does not return at least a 'missing' for them, the class does recognize this now and generate and add dummy results for the missing pages.
Spezial:Permanentlink/72481024#Dein Bot: komische Einträge Notification links (section matching); strange behaviour. When using wiki API to check sections, both methods are now used always, instead of only when the first returned nothing and the longer output is used (since the other might not be complete). This should prevent problems with pages like Spezial:Permanentlink/72550929 with any kind of include-tags. This method now is slower since always 2 calls are made, this should be improved. (bot run now takes about 1.5h instead of about 1h before; 50% slower...)
Creation of a stable release v0.2.0000 based on bug fixes, new feature and improvements in current semi-stable release v0.1.0013. Testing of this version and update of bot software on toolserver to this new version. Extensive testing of all new functions and whole code like output of times (correct timezone setting) and others.
Setup of new CRON-job for compression of history once all 14 days.
(toolserver)
MailerBot created. This sub-bot is very experimental at the moment; it does it's job for me, probably/hopefully for you to, but do not expect miracles at this current state (but ideas, hints and bug reports are still very welcome).
because the bot runs every day at least for 1 hour, so the load must not be too high! If it is, try to delay the expensive calls and/or contact the toolserver admins what to do, because 'nice' is already in use. The load of history compression has not to be checked, since the script runs not that often.
Result:
drtrigon@nightshade:~$ /usr/bin/time -v python /home/drtrigon/pywikipedia/runbotrun.py -cron
Command being timed: "python /home/drtrigon/pywikipedia/runbotrun.py -cron"
User time (seconds): 32.23
System time (seconds): 2.86
Percent of CPU this job got: 1%
Elapsed (wall clock) time (h:mm:ss or m:ss): 46:18.79
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 42316
Voluntary context switches: 204822
Involuntary context switches: 3707
Swaps: 0
File system inputs: 0
File system outputs: 1816
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
should be ok with 1% CPU load.
(toolserver)
B44
For several pages related with MerlBot the bot was not able to retrieve section (heading) info. Those issues appeared on following pages:
and were related to <noinclude> and <onlyinclude> tags, and have been solved. But the pages are handeled like templates because of 'action=parse&text={{%s}}__TOC__&prop=sections' is used for recognition. Therefore it could become necessary in future to use a more sophisticated way; get page content, preprocess (handle <noinclude>, <onlyinclude>) it, pass it to 'action=parse&text=%s__TOC__&prop=sections' and then postprocess (handle the rest like <pre> and others) the result.
dtbext_wikipedia.p
F45
List of parameters/options enhanced:
checkedit_list: each User has the ability to add additional patterns for pages that should be monitored by the bot.
Spezial:Permanentlink/62158091 / 25. entry; the brackets '[' and ']' have to be replaced in some parts of the output (namely the url part of the links).
sum_disc.py
B45
Bugfix for:
Spezial:Permanentlink/62087384 / 2nd last entry; is reported despite the fact it is signed/not necessary to report. The problem was that a user used english page links e.g. 'User:...' in its signature, those describers were added now.
sum_disc_conf.py
From id 44 splitted into B?? and F?? for simpler counting.
altsign_list: especially for BLUbot introducted parameter to give alternative user names to check for signature, e.g. you have a representative or something else.
sum_disc.py, sum_disc_conf.py
40.2
Bugfix for:
Spezial:Permanentlink/61457254 / last block, 1st entry; the last part of '_checkThData()', check for text after signature recognizes the '|}' as additional text, code to prevent this was tested but not activated beacuse the better way would be to change the welcome template and suppress additional newlines at the end.
The bot reports redirects (created by users) as new discussions (which of course are non-existent). Maybe redirects should be resolved and the resulting pages checked against the ignore-list?
This change was mentioned by BenzenC6H6 09:20, 5. Jul. 2009 (CEST) (probably a hamster?).
dtbext_wikipedia.py, dtbext_query.py, sum_disc.py
17.2
Wanted to change the history format such that each time, at each bot run the last history file entry is read but contains the full actual history. This data is updated and checked for changings since last run, the actual version history for each page is retrieved and saved - this would serve a simple programming interface, however it uses way to lot of hard drive space (and memory). Now using the old incremental format, but with the data updated and checked for changings since last run and 'compressHistory' does no additional relevancy check, just rewrites same history content (bot behaviour keeps the same after compression). So fast compression was removed completely.
Dogma: 'compressHistory()' must not do any kind of relevancy/signature check, and not alter the bots behaviour.
At bot run (not compression) all discussions are checked for signature and if it is completely missing the discussion for this page is closed and it is removed from history.
Additionally 'rollbackHistory()' was implmented to make the bot to use older history entries only.
Different pagestates introduced to make the new history handling practible:
_PS_warning: serious or no classified warnings/errors that should be reported
_PS_changed: changed page (if closed, will be removed)
_PS_unchanged: unchanged page
_PS_new: new page
_PS_closed: closed page (remove it from history)
_PS_maintmsg: maintenance message
Split of 'self._news' into 'self._news' (_PS_changed, _PS_new, _PS_maintmsg) and 'self._others' (_PS_unchanged, _PS_warning) and furthermore parallel (to 'self._news') updating of 'self._history' (_PS_changed, _PS_new, _PS_maintmsg, _PS_unchanged, OHNE _PS_closed) which is saved in the end to the history file.
History should be dropped at least all two week, because it is bigger now (could be zip-ed)!
sum_disc.py
35.2
Method of choice for wiki content access is to use the API and retrieve pages in bunches (of e.g 10 pages). Dumps are very fast too but not available on a daily base (all 2 days only), look at http://download.wikimedia.org/dewiki/ also (Benutzer:MerlBot claims to use dumps). So switched to handling of page read access with API (it looked to me that the recent rev. of bot-framework have switched partly to access over API, but that is wrong; it are a few commands only, mostly user or config related), this was already implemented with different methods in dtbext.wikipedia.Page().get(). Now it was implemented for simultaneous read of a bunch of pages with prop=revisions (rv) in dtbext.wikipedia.Pages().get(). But this has a small drawback, the parameter/option 'directgetfull_switch' becomes a kind of useless or obsolete (because it is faster to re-read single pages per block, then splitting the block and all handling related with this), therefore removed parameter/option:
directgetfull_switch
It is important that the original and the new page read methods deliver the same page content!
look also at id 17
dtbext_wikipedia.py, dtbext_query.py, sum_disc.py
21
Merged with most recent pywikipedia-framework 6977 and testing of new functions, as well as update of wikipedia.py (because of lines 725-728). The bot reports now on every run the most recent framework rev. because of verbosity and information (download via svn could be automated, look also at Benutzer Diskussion:Benzen#Fragen wegen Bot-Modifikation). The extensions needed for the bot are now techn. adapted to the framework and packed into one single package with same structure like the framework:
wikipediaAPIdtbext.wikipedia and dtbext.pagegenerators (analogue to the framework)
wikipediaAPI.ContributionsGendtbext.pagegenerators.UserContributionsGenerator (is now a generator and returns a list of pages instead of a list of tupels)
wikipediaAPI.BacklinksGendtbext.pagegenerators.ReferringPageGenerator (as above), furthermore dtbext.wikipedia.Page().getReferences() with API implemented. If the API is used for call is not related to the generator but to the given page-typ. The new generator supports setting of limit/number additionally.
wikipediaAPI.Page().getFull(), wikipediaAPI.Page().getParsedContent() and wikipediaAPI.Page().get() (the same as wikipedia.Page().get()) dtbext.wikipedia.Page().get() with mode='full' , mode='parse' or without any setting (default)
wikipedia.sectionencode for bot messages could be possible to solve the old special char issue
dtbext.wikipedia.html2notagunicode introduced (as addition to 'Unicode library functions')
clean_sandbox2.pyclean_user_sandbox.py
dtbext.query introduced
(various)
35.1
Introduced bundeling of API calls. A new class 'Pages' performs all actions on a set of pages simultaneously. Currently page version history information can be retrieved this way, this can be enhanced on other functions too.
look also at id 17, 35.2
wikipediaAPI.py, sum_disc.py
Full compression (not fast) improved by adding a part that removes all old/removed headings from history page entries.
wikipediaAPI.py, sum_disc.py
New history format (v5) introduced; use dict (again) is the one and only intelligent/correct way to hold unique data
Conversion of old history formats in '_getHistoryPYF' during read-in, instead of different places spread all over the code. This causes all history data to be keept in up-to-date format, now.
Dogma: history format for headings (with checksums) has to be a dict, with unique keys. (if not possible to make unique, then add a timestamp or something like this)
sum_disc.py
Heavily improved parameter/option handling, all internal used user specific bot parameters/options can be modified by the users. The behaviour either to overwrite or preserve the defaults can easily be configured internally.
sum_disc.py, sum_disc_conf.py
18
List of paramters/options enhanced:
ignorepage_list: here you can add pages to ignore, to this list added are your own discussion pages. if you do not define this parameter your archiv pages are used by default.
sum_disc.py, sum_disc_conf.py
17.1
Speed-up by:
pre-compile of all often used and static regex (probably do some more later, if it makes sense)
Dogma: frequently used regex must be pre-compiled.
wikipediaAPI.py, sum_disc.py
28
Clean the code by removing all unnecessary 'try' constructs, replace '(... == None)' contructs by the shorter (and cleaner) version and add documentation.
look also at id 26, 27
sum_disc.py
32
Compression could be done by comparing against the backlinks. This was implemented and tested, here the observations: the original method depends on '_SearchForSignature(...)', the alternative method on a properly configured 'backlinks_list'. if properly configured the alternative mode is a lot faster, but it depends on this config, whereas the original method works always! the statement all pages listed on backlinks have currently a signature is correct if 'backlinks_list' is properly configured (how to check this?)
sum_disc.py
27
The checksum checker in '_checkThData(...)' improved (no need for 'try' structure anymore) and the different modes documented. Mode v1 was removed because it is very old, if there is any history entry with v1, then it is ignored, the page is reported and logged again. This should cause not that much problems (since there are very few or no such entries existent anymore).
sum_disc.py
31
Checkout of API functions/calls, here some examples:
meta=userinfo (ui) with 'hasmsg': gives only a hint if there is a new message or not (already implemented in bot framework) but how to set it read after loading the page to make the message disappear is still left unclear / function not needed at the moment
meta=allmessages (am): can be used to translate some describtions and other things from English to this wiki / function useful
meta=siteinfo (si): a lot of site, wiki and server info / function useful
action=paraminfo: obtain information about certain API parameters / function useful
getbacklinks_switch: read the backlinks to user discussion page and check them additionally
wikipediaAPI.py, sum_disc.py
29
All pages 'Wikipedia:Löschkandidaten/<DATE>' are read twice (second time with 'getFull()'), therefore a parameter/option was introduced:
directgetfull_switch: read all pages with 'Wikipedia:Löschkandidaten/\d' at first attempt with 'getFull()'
look also at id 17
(This option may later become obsolete, e.g. when updating to new bot framework; if the default read method 'get()' is changed)
sum_disc.py
26
Bot should run without any use of 'fall back' mode in '_getThContent(...)' therefore this obsolete, old and unreliable mode was removed but replaced with a warning message (just in case).
All those are related with the different history formats and the way they were read in and updated. All older formats are converted now during read and in to v5. Took from Version 0.1.0014.
sum_disc.py
Bugfix for: Spezial:Permanentlink/61457165 / 1st entry; head recognition out of sync, added another check for validity (to decide to use full read mode). Took from Version 0.1.0014.
wikipediaAPI.py
Bugfix for: html tag heading recognition; tags with additional values are now recognized too.
wikipediaAPI.py
Bugfix for: heading recognition; e.g. Spezial:Permanentlink/61381636 / last block, 1st entry; because of some <nowiki>-tags the function had some problems. All '=' within those tags are substituted by '=' as work-a-round.
wikipediaAPI.py
Bugfix for: re-reported discussions; e.g. Spezial:Permanentlink/61311960 / last entry orSpezial:Permanentlink/61309424 / 1st entry of last block; this is related with the different history formats and the way they were read in and updated. Some older formats are converted now during read and in that way the data can be updated properly.
sum_disc.py
Bugfix for: twice appearing headings; the API returns wrong links they are not complete, the numbering (from 2 upwards) is missing. So a fix for this was implemented now the bot should really be able to get all head links correct.
Bugfix for: Spezial:Permanentlink/61105710 / 1st entry of last block; the page was processed without any heading info, in fact it could not retrieve the headings (sections). Added an additional/alternative method if the first fails.
look also at id 28
wikipediaAPI.py
35.4
Bugfix for: Spezial:Permanentlink/61106764 / 2nd entry; the headings got out of sync, in the 2nd half it was shifted about 1. The problem was that headings recognized by wiki parser has to start at the first char of a line!
look also at id 28
wikipediaAPI.py
5
testing of new 0.1.0013 release. upload of it and rearrange data on toolserver.
copy 'local:pywikipedia-devel_0.1.0013/' to 'toolserver:./pywikipedia/'
copy 'toolserver:./pywikipedia/' to 'toolserver:./public_html/source/pywikipedia/'
copy 'toolserver:./pywikipedia/' to 'local:toolserver/pywikipedia/' and 'local:toolserver/public_html/source/pywikipedia/'
local, toolserver
15
it would be a good idea to use https://jira.toolserver.org/secure/BrowseProjects.jspa the Toolserver issue tracker if the amount of bugs reported and to handle, exceed a certain number, but for the moment it is not needed. the problem is you cannot add a new project as normal user, so the system here has to deliver this service.
had issues with creation of BOT-MESSAGE containing some special chars (e.g. ':'). the algorithm should now be more stable, but it estimates the correct heading only, so could still make some problems.
Dogma: use as simple bot maintenance message texts as possible.
if such a message apears again, the bot now read its discussion page Benutzer Diskussion:DrTrigonBot and prints/show the content to output (log file). that informs you about the message and should lead to supress further display (of the same old message).
if the bot is executed by CRON job on toolserver the preprocessing/filtering of log file output does not work properly. have added another regex to the output handler, hope that helps.
runbotrun.py
10.3
purge page cache through wiki API according to example http://toolserver.org/~drtrigon/wiki-purge.html (has to be deleted from the toolserver now) implemented as wikipediaAPI.Page(...).purgePageCache(). the issues occured here earlier have disappeared or were due miss-programming/-understanding of the API by myself.
wikipediaAPI.py, sum_disc.py
10.2
the render given in http://svn.wikimedia.org/viewvc/pywikipedia/trunk/pywikiparser/ looks to be (very) old; the newest contributions are Modified Mon Oct 29 06:41:05 2007 UTC (19 months, 1 week ago) and the wiki API seems to do a quite good job. so the bot is using the API at the moment.
wikipediaAPI.py, sum_disc.py
11
because of improvements in 10.1 before the bot is now able to detect level 1 headings (through a work-a-round, but it is possible) with the old it was nearly impossible. so level 1 headings are taken into account except the bot has to fall-back into the old mode so the wrong display of discussions in front of level 1 headings (on problematic pages such as Löschdiskussionen bei Benuterseiten, WP:FzW, WP:Auskunft, Portale and others) should stop now.
wikipediaAPI.py, sum_disc.py
10.1
wikipediaAPI.Page(...).getSections() improved; now it is on a good level and able to find and synchronize the subsection headings in pages
wikipediaAPI.Page(...).getFull() introduced; to help ...getSections() it uses 'expandtemplates' API and therefore should be able to give a low level output of wikitext on pages, however it seems to hide HTML comments for example...
_checkRelevancyTh(...) improved with help of ...getSections() but it still has full fall-back capabilities.
wikipediaAPI.Page(...)._getParsedString(...) introduced and wikipediaAPI.Page(...).getParsedContent() improved; they should work now.
_parseNews improved because of the two mentioned before it was possible to replace _renderWiki(...) (and with it _renderWikiTempl(...) too) with _getParsedString(...) and they are useless now and can be deleted.
wikipediaAPI.py, sum_disc.py
7
'_setUser(...)' and '_getUsers()' used both two seperate parameter/option handling mechanisms. since the entry in Benutzer:DrTrigonBot/Diene Mir! without and parameters in brackets '{...}' offers already a config option; you can specify any target page within your userspace. this option and the "real" options in brackets where handeled in 'self.userListInfo' and 'self.param'. those two mechanisms were now merged and the target page in 'self.userListInfo' is added to 'self.param' as 'userResultPage'.
Dogma: 'self.param' is the only option/parameter handling mechanism (especaly for params changable by all users through wiki)
sum_disc.py
6
config management/system for running bot in seperate file; all options (global vars) moved there. the variables could be named nicer (more self-explainig) but the external options names (parameters that can be set within Benutzer:DrTrigonBot/Diene Mir!) must not be changed! never!
Dogma: the external options names (parameters that can be set within Benutzer:DrTrigonBot/Diene Mir!) must not be changed! never!
checkedit_count: how many recent edits should be checked (mainly for use with BLUbot)
reportchanged_switch: should changed (and already as new reported) discussions on change be listed again (mainly for use with BLUbot)
sum_disc.py
12
format/system of saved checksums had to be changed once again (3rd or 4th time) because of the dict used with subheadings as key was not unique (all subheadings on a page may have the same name); now the bot is using a list again, but a list of tuples (there are still some issues open with subheadings, this may make another change necessary...)
sum_disc.py
4
add interface for history's compression; was done initially but does not make sense since the permissions of user 'apache' will not allow to to re-write the history. the good possibilities are to do it by hand (manual) with a toolserver login (from time to time) or setup a e.g. monthly or weekly CRON job
Dogma: history compression should be done by hand or CRON job (if not by hand, do auto backup)
panel.py
3
improvement of admin log interface: 'filelist' and 'checkfiles' are not necessary, the whole path can be generated only once just before the call of 'os.remove'!
panel.py
2
instead of using 'runbotrun.py -auto', now a CRON job was created on toolserver, this should in order take care on the bot timing and be able to deal with server reboots. with this action all issues related with this are solved. call is of bot is 'runbotrun.py -cron'
runbotrun.py
1
admin interface added for removal of old log files instead of using cyclic logs (5 files e.g.)
Dogma: admin interface is used for log file removal - it's link is not published
