From ee507c4d53c6e758d7df8b3be18803a5c3ebbde3 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Mon, 16 Sep 2019 17:24:35 -0500 Subject: [PATCH 01/10] Adjusting the bodge to fix the multiple comment bug in issue #1 --- CHANGELOG.md | 63 ++++++- FAQ.md | 2 +- README.md | 2 +- ROADMAP.md | 322 +++++++++++++++++++++++++++++++- utils/archiveAndUpdateReddit.py | 29 ++- 5 files changed, 407 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8935512..848f69e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ -# CHANGELOG: Python Helper Bot Version pre Alpha A0.3.00 +# CHANGELOG: Python Helper Bot Version pre Alpha A0.3.02 All notable changes to this project will be documented in this file. The format is loosely based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). @@ -18,9 +18,68 @@ Dates follow YYYY-MM-DD format +## [A0.3.02] 2019-09-16 + +Official. + +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +This project is not currently looking for other contributors + +#### Big Picture: What happened, what was worked on +The bot was commenting multiple times on the same post if a ratelimit error occured, and a bodge is now in place to check this behavior, the previous page checked its own comment history to minimize api calls (assuming grabbing the most recent 2 bot comments was enough). However, when reddit's servers have an issue it became apparent that the bots user history and the submission's comments do not necessarily match up. + +Now the bot checks the submission for multiple comments, and also automatically assumes that the comment went through just in case. + + +#### Added +#### Changed + - Bug Fix [Issue 1](https://github.com/CrakeNotSnowman/redditPythonHelper/issues/1#issue-473053676): To address Issue 1, in Archive and Update Reddit: comment_duplication_by_ratelimit_check() which takes in reddit and the submission, and now grabs an instance of the submissions top level comments. It then itterates through the comments checking for a match with its own username, and if that is found, it adds it to a list of its comments on that submission. In the event that multiple comments by the bot are found, it logs an error + - The bot then ignores the previous bug fix and assumes the comment went through. This change should be reverted eventually, but until another server issue occurs, I can't see how the bot will behave and as such have to assume that despite the 500 error, the server will show the comment eventually. +#### Deprecated +#### Removed +#### Fixed +#### Security + + +### Main + +### rpiManager.py + + +### Util Libraries + +#### archiveAndUpdateReddit.py +#### botHelperFunctions.py +#### botMetrics.py +#### botSummons.py +#### buildComment.py +#### formatBagOfSentences.py +#### formatCode.py +#### learningSubmissionClassifiers.py +#### locateDB.py +#### lsalib2.py +#### questionIdentifier.py +#### rpiGPIOFunctions.py +#### scriptedReply.py +#### searchStackOverflowWeb.py +#### summarizeText.py +#### textSupervision.py +#### updateLocalSubHistory.py +#### user_agents.py + +### Tests + ## [A0.3.01] 2019-07-26 -In Progress +Official. ### Contributors Keith Murray diff --git a/FAQ.md b/FAQ.md index 1cefc2c..a4387a1 100644 --- a/FAQ.md +++ b/FAQ.md @@ -124,4 +124,4 @@ When I get to that point, I'll probably just have folks tackle specific elements They seem cool. I've got no problem with them. -#### Version Pre Alpha A0.3.00 \ No newline at end of file +#### Version Pre Alpha A0.3.02 \ No newline at end of file diff --git a/README.md b/README.md index 0083562..05d08b9 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# Reddit Helper Bot: Version Pre Alpha A0.3.00 +# Reddit Helper Bot: Version Pre Alpha A0.3.02 pythonHelperBot is a reddit bot built to analyze r/python post and determine if they're better suited for the r/learnpython sub. If they are it suggests that the user post to that sub rather than to r/python. diff --git a/ROADMAP.md b/ROADMAP.md index 4012cea..0940cc4 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ -# ROADMAP: Python Helper Bot Version pre Alpha A0.3.01 +# ROADMAP: Python Helper Bot Version pre Alpha A0.3.02 Future expansions are considered in this file. Their presence is not a promise that they'll exist, but rather this file serves as an early outline of features this project hopes to add, as well as changes in directions @@ -21,7 +21,9 @@ Dates follow YYYY-MM-DD format -- Susan Calvin in "I, Robot" by Isaac Asimov -## [A0.3.01] 2019-07-26 + + +## [A0.3.02] 2019-09-16 In Progress ### Contributors Keith Murray @@ -320,6 +322,322 @@ Think about using a subset of highly matching SO posts code to OPs source code a Most likely this is especially useful with syntax errors and stack traces. +### Generalizing the bot: +These are features which an ideal bot-mod would have, but which are not directly linked to a question-answer-and-redirector bot like u/pythonHelperBot (as of mid July 2018) + - sub_Toxicisty_Score(): + Alternatively a friendly score. Bit ambigous, and doesn't immeadetly fit into the bot, but just a measure of how kind or standoffish or toxic a sub is. Certain communities tend to forget that not everyone knows everything, and it'd be nice to avoid recommending those subs. + + - blog_Spam_Flagger(): + This is actually a large but distant future goal for the bot. There's often complaints about blog spam on the python sub, and it'd be nice to have a programmatic way to define it. Even if the spammy site sees the definition, and works around it, the definition can either be altered, or the work around can be allowed. Most redditors want to see good content, so the best way around an ideal blog spam filter would be to have variable, high quality content. In which case everyone wins. Using that idea, we can start to outline the basic components of what blog spam might be. + + High quality content is safe. High quality with respect to the python sub is probably some function of what generally does well + + Low quality can be caused by a few reasons: r/python is not the proper sub for that: ie questions + It was recently posted: this is probably best defined as content theft, though repost is a common name for it. + + I'm tired, I'll come back to this. + +### Tests + +## [A0.3.01] 2019-07-26 +Completed. +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +### Short Term Roadmap + +Add a duplicate comment check just in case the ratelimit error posts anyway. + +#### Add + - Migrate most files to usb/usb-sata drive to host. Watch power requirements. + - With light archive, build a 'save state' and 'load state' function so records on new posts aren't lost during reboots. + - [X] (NO CHANGE MADE) Verify the key phrase response checks for self post v image post. Ok this code is written and in learningSubmissionClassifiers, but after looking through the older versions of this, it never was a requirement. And it's usually pretty spot on. The code is commented out, so think on it for a while. [Update] While it comments on non self posts, these are pretty accurate and I don't view this as an issue. I'll allow it to continue to post on non self posts if a key phrase is used + - loggingSetup.py: a module to be imported first by rpiManager and main, which sets up the logging format so the program can be called by either module on any system and initialize in the same way + - botHelperFunctions/botMetrics: Add ram usage check and log it in every 'awake' cycle of program, to try and tease out any possible MemeoryError's kicking up after long periods of time. Might as well save other diagnostics info, maybe call this from the hearbeat thread so it becomes a better representation of runtime + - rpiManager.startupSwitchFlag(): + Moving this higher up. + A function which polls one of the gpio pins to see if it's low (normally high) (or flipped levels). That pin will be tied to a jumper or switch. When set, the bot will not turn on at boot, allowing for simpler diagnostics, updates, etc. + Also a pull origin master from github state would be incredibly useful + - Verify reddit post logs by grabbing most recent bot comments. This should reduce risk of two computers commenting on the same post which could happen if databases are de-synced and one computer is not in quiet mode because I typed in the wrong command. Opperator Error Risk Reduction. Only the most recent x+buffer hours of interaction are needed. This does not protect from multiple posts by the same user but should prevent multiple comments on the same post even if the bot runs on different computers. + - Full Test Suite: PRIORITY (Ok it'll have to wait until posts have been archived, but it'll still be nice) + - Archive Posts: PRIORITY + Solidify what values to save, and what to save them with. Probably build an SQL to XML or JSON exporter for third party testing. + - botMetrics.measureUserReaction(): + A function focused on seeing if a user did in fact go to + r/learnpython after the bot made its suggestion. Currently built (kind of, the praw wrappers messed it up a bit), now need to add + functionality in main.py to use it + - Continue Documentation in functions, add documentation files too + - A queue which has inputs added to it by functions, and removed by the raspberry pi gpio handler, allowing the bot to change LED status based on what it's doing. Similar to the twitter event bot design (sepperate project). + - Deprecate karma scatter plot: or change it post all posts in the past week to minimize size. It's no longer useful or very interesting. Maybe activate it once a week or something too + - Consider creating praw rewrapper (prawRegift) to hold all praw focused wrappers and sepperate it from the phb functions. This will make it easier to have the same protections on other bots as necessary. + - botMetrics.predictUserReaction(): A function to go through users comment history, look at the parent comments, and from that gauge how the user will respond to the bots help. In the future adjust how the bot replies based on the predicted responsiveness. For now, it'll just build an archive of users responses to previous comments. + + + +#### Change + - Move NLP functions from NLTK (used in many files) to a buffer module, allowing for simpler, universal changes to be made. For example, if there is a better POS tagger than nltk.pos_tag(sent) then we can easily switch to that. Right now NLTK is used over a fairly large filespace making adjustments of that sort difficult. + - Review all my logging notes. See what should be dropped, changed, etc. + - Make sure the bot defaults to commenting about formatting even if there's no code present + - When errors occur, or a shutdown button is pressed, attempt to save current reddit info into a temp file. on startup, load that info in then connect with praw to expand the knowledge base. + - The archive and update reddit module has become unwieldy. It should be broken into two modules, one which handles the recasting of the praw classes, and one which deals with them as needed. + + +#### Deprecate + +#### Remove + +#### Fix + - Standardize function name style. Either underscore or camelcase, just not both + Probably preferable to use underscore, the despite camelcase being faster.. + +#### Security +#### Consider + + + +--- + +### General to Long Term Expansion + + - Develop terms for a walk away condition. Either End of active development and the bot remains online, end of active development and death of bot, or end of active development and project is passed on to others. Terms will almost certainly be changed constantly and the project grows and evolves, but it's nice to have an idea of what I consider to be a "complete" project. + + - Numbering system for items in roadmap to clear up what's being worked on and what is completed from an outside perspective. A master numbering system probably is a good idea, vX.X.XX[a,c,d,r,f,s,co]XX, following version, section, and specific roadmap suggestion number. But That seems bloated and unnecessary. (Maybe this isn't worth while, maybe it is and will help catch things in the changelog. Probably wont be seriously considered until alpha) + + - summarizeText.loadEnglishModel(sourceDataPath=paths["prebuiltEnglishDB"]): + Prebuild and pickle the output tdm of + summarizeText.buildModelFromDocsInFolder(sourceDataPath=paths["englishDB"]) + so the raspberry pi doesn't have to hit memory errors in in main.startupBot() + load the prebuilt database and build it if the prebuild database does not exist. + Or just build a custom compression scheme and load that instead of using pickle. + + - test.EvaluatePost(): + Given recent restructuring, this should be much easier. Take a post given a post id, then run it through the classifier where the exitpoints are turned off from the functions, forcing it to classify the post in full. Because the praw wrappers are in place, there shouldn't be a concern about forcing full evaluation any more. This can be considered to be half completed: the silent mode the bot has helps evaluate posts. + + - Reply To Common posts: + Build semi scripted replies to frequently asked questions (probably largely pulled from the sidebar, since that's how the side bar gets populated) + This includes "How do I install python" and "Is learning python worth it?" (Aka "Why learn python?")--A quality version of this is a full research task. Build off of internal sentence ordering models + soft skills. Pull text from comments on these posts to build up scripted reply. Look into adapting this (or maybe starting with this) to classifying the posts so their types can be diplayed as tags. + + - local Flask website/dashboard to monitor the status and logs of the bot in realtime. Low priority. + + - Log processing: a set of functions and visualizations to process the log files for various useful tidbits. Something nicer than grep + + + + +### Main + - alreadyAnswered(): + Parse through OPs comments on the thread, and search for text that implies the question + has been answered. Adjust comment on submission accordingly, probably to say, "Next time + you have a question like this, consider using r/learnpython" blah blah blah + + +### rpiManager.py + - update the commented gpio naming and numbering list + - update grab-from-github functions + - add a queue to work with rpiGPIO for LED displays for various tasks + + + +### Libraries: + + +### archiveAndUpdateReddit.py +Most of the 'light' archive functionallity is currently being built out, rendering a large chunk of this section of the roadmap either completed, in progress, or dismissed. It is also no longer in the realm of 'long term'. + +[x] The big set of functions necessary in this module are database creation, and update functions. +There might be two databases: one of just posts, and another comprising of posts, and comments. + +ARCHIVE FUNCTIONS +TODO: Saves it into sqlite3 table after it's passed the time threshold to "not in +use" + +"not in use" is probably going to be defined as 8 hours. Past that point the +post will be either 'successful', 'mild', or 'unsuccessful', defined as +x >= 8 points, 8 > x >= 1, 1 > x + +Adjust it so 'not in use' is not defined as 8 hours, but instead defined as +a varible, which changes based on the time of day (either defined by utc or +cdt--cdt being my current local time) that the posts was made. 'Late at night' as +defined by the time where the fewest users/ r/python 'actions' (posts, upvotes, +comments) are made, adjusted according to the day of the week and or holiday +(unlikely that this bot will need to be that specific) the post is made on. + + r/learnpython is another source of data: + +it will act as a source of useful questions, and will allow the bot to direct +users to other reddit based questions rather than simply stack overflow (this +distinction should help allow the bot to be generalizable) +posts between 3 and 8 upvotes will be determined as 'basic questions' and +will be used as suggested solutions if the similarity between the new r/python +post and the old r/learpython post is greater than some threshold + + +### botHelperFunctions.py + +### botMetrics.py + - predictUserReaction(): used for the bot gauge how receptive the user will be towards a r/learnpython suggestion. If it predicts combative, it'll make it's comment short and to the point. If it predicts receptive it'll expand on highlighted points such as formatting. If neither it'll contiue as is. + + - measureUserReaction(): +to see if redditor does post to r/learnpython. The post will have to be strongly similar +to their r/python post, and be posted not long after the python sub post + +- questionAndAnswer(query): to attempt to reply to semi-scripted questions + +- buildConfusionMatrix(): to measure performance + +#### Confusion Matrix Traits +##### True Positive: +[The bot has commented,] And +[[Either a mod has removed the post due to 'learning'], +Or [the redditor posts their question on r/learnpython]] + +##### False Negative: +[The bot did not comment after 8 hours] And +[[Either a mod has removed the post due to learning,] +Or, [[someone else has commented r/learnpython] and [has greater than 2 upvotes after 8 to 24 +hours after commenting,]] +Or, [the user posted their question on r/learnpython]] + +##### False Positive: +[The bot has commented],And [has less than -1 comment karma after 8 to 24 hours,] +And [[the post is not removed due to learning within 8-24 hours] or [by mods recent +activity plus some threshold.]] +And [[the user does not make a similar post to r/learnpython within a timespan of 8 +hours] or [4 hours after their next user activity monitored for no more than a week]] + +##### True Negative: +[The bot did not comment after 8 hours] And +[[No mod has removed the post using a reference to 'learning' after 8 to 24 hours] or +by mods recent activity plus some threshold.] And +[[No commenter has post made a post which contains 'r/learnpython'] And [has more than 2 upvotes]] + +##### Fuzzy: +All Else. +This class will be either require human moderation to place into the confusion matrix, Or +will be used for other classificaiton, such as "Blog Spam". It could be that topics placed +in this area can be re-examined and labelled, helping the bot generalize preformance in +other areas + +### botSummons.py + - Finish makeFormatHelpMessage summons +### buildComment.py +### formatBagOfSentences.py +### formatCode.py + - formatCode.py: Cleave sentence from comment and first line of code from one another + - formatCode.py: Using rewrapClassifications output, check to see if any indentation is present for lines that have been classified as code. If >5 lines of code are present and none of them have indents, classify block as "The reddit text editor royally screwed this one up", adjust comment to say it's unlikely that the code has been indented properly, and enter the special fixer. + - formatCode.reformatFromHell(): Read in all previous code. Read in current line. If rfh classification Adds indent: current line is a child of the previous line. If it is the same indent level, current line is a sibling. If it is minus indent, line is a sibling of the previous lines parent. + - Previous code is stored in a tree like structure + - Leverage sentence ordering ideology to say given the current line and the previous state of the code tree, which level of node in the tree should I be + This should be an area of linguists where there's plenty of work already completed, look for it. I think Nevil-manning sequitor addresses it briefly, look at that+cited by for other work in the area. + + +### learningSubmissionClassifiers.py + +### locateDB.py + - load in path data from a prefernce file, and or take it as input that way the path isn't + 1. Hard coded and + 2. Hard coded in the module + Generic is better if it's generally useful. + + That said, "check_though_these():" is a pretty good and simple function to move out of "locateDB.py" and into main.py + + - Call a function in this library to recast folder/file calls to the correct os format. Or just redo it everywhere in the code. + Whatever works best + +### lsalib2.py + +### questionIdentifier.py +It'd be nice to use stack overflow's user submissions and r/learnpython's +submissions compared to 'successful' r/python submissions to build a 'programmers +question' classifier (and expand the classifier to blogspammers). This would +make it generalizable so posts which are questions or requests ("HELP ME CODE") +are directed to r/learnpython, posts which are clearly for click/ads are commented +on as such, and good posts are 'ignored': allowing redditors to act on it as they +choose. This is not an easy goal to acheive and is incredibly arbitary. but there +are still certain factors which can be measured and acted on. + +This will probably leverage a stack overflow search engine and compare n results +with k or greater similarity. + +### rpiGPIOFunctions.py +### scriptedReply.py +### searchStackOverflowWeb.py + - Scrap and rebuild with approved api and bound it to search for results between + local database build date and present day. Not important until after local copy of SO + is up and running +### summarizeText.py + - Improve the english language model for topic modeling, and focus on programming topic modeling. +### textSupervision.py +### updateLocalSubHistory.py +### user_agents.py + +### OTHER +(This is all functions that don't have a clear parent module) + + - moqaProgram + + - ELMO/BERT programs + + - reformat_User_Code(): + a function to identify python code blocks that aren't properly formated, and auto format the code + for other reddit users. Might live it its own module. + Currently being worked on. + + - Leverage reformat user code with automatic Q&A: Use classified code regions to match SO code regions, classified text regions to match SO text regions. Hopefully this improves the search engine and cuts the risk of added noise by a text to code block increasing precieved distance between the user query and the SO database post. + Next If a majority of highly matching SO posts have sample code in the question, but the reddit query does not, strongly suggest adding the example code that caused the issue to the next itteration of the query. + + + - question_topic_Modeling(): + This is going to take a few parts. + - Identify all related learning subreddits: + + - Model the topics of stack overflow questions. + + - Model the topics in the learning subs + Do network analysis to find the most active sub that addresses a topic: probably pagerank since it's simple and it works. It doesn't need to be state of the art, and if it can run on the pi, that's even better + + - Next take in the question, extract topics, feed the topics in the network, identify the sub that will get the best answer fastest. This means there also has to be some knowledge of the subs activity score + + - sub_Activity_Measure(): + Or score.. + This will probably return some arbatrary number that only makes sense in the context of other measures + It might be a function of: + The distance between the top 25 posts on Hot and the top 25 posts in New, where 'top' refers to + reddits ranking. + The number of comments and the absolute value of karma of those comments + the number of unique users in those 25 posts + The time between each activity + + Comparing the intersection of hot to new posts shows a glimps of how active the sub is without requiring the bot to look at the sub at multiple times. + + This function would be useful with the question_topic_modeling() function and wouldn't need to run frequently. Though over multiple runs, it would have a solid understanding of how active a sub is at different times of day, which might encourage the bot to direct a user to a learning sub that is + active at that time. + + + - Auto Reply to common questions (Functional FAQ as it were) + (This is probably going to be an early test of soft skills) + * ["Possibly wanting to learn Python, is it worth it?"](https://www.reddit.com/r/Python/comments/917zxd/) + + - Use Automatic Sentence Ordering to construct the bots autoreply, reducing the mess of the code there. Should be mildly simple (ha, sure...), and allow for much more flexible commenting. Target is to have a defined intro, a 'bag of sentences' for the body, and a defined signature. The 'mildly simple' notion is built off the idea that there will be little the program can do incorrectly with that scaffolding. Look at two metrics: absolute sentence ordering, and new paragraph insertion. Maybe train on a ton of readme's, or wiki data for the new paragraph insertion. + + +#### Question & Answer +Resources to draw from: + - Stack Overflow (Primary) + - Python Docs (Secondary) + - Python Blog Posts (Out of Focus) + - Scraped Github Code (Out of Focus) + +Think about using a subset of highly matching SO posts code to OPs source code and using bayes in a MSAlignment fashion to guess on solution. +Most likely this is especially useful with syntax errors and stack traces. + + ### Generalizing the bot: These are features which an ideal bot-mod would have, but which are not directly linked to a question-answer-and-redirector bot like u/pythonHelperBot (as of mid July 2018) - sub_Toxicisty_Score(): diff --git a/utils/archiveAndUpdateReddit.py b/utils/archiveAndUpdateReddit.py index 8dcc193..fc4fb90 100644 --- a/utils/archiveAndUpdateReddit.py +++ b/utils/archiveAndUpdateReddit.py @@ -1135,6 +1135,8 @@ def is_connected(): def comment_duplication_by_ratelimit_check(reddit, submission): ''' + Repurpose this to check the submission, not bots history + This is a bodge check to see if the bots comment went through and was posted despite catching a ratelimit error which directed the bot to try again @@ -1142,19 +1144,31 @@ def comment_duplication_by_ratelimit_check(reddit, submission): See issue https://github.com/CrakeNotSnowman/redditPythonHelper/issues/1#issue-473053676 ''' - timeDelay = 5 # seconds + timeDelay = 60 # seconds logging.debug("Got Rate Limit Error, Waiting and checking if comment went through") time.sleep(timeDelay) - phbot = get_redditor_by_name(reddit, 'pythonHelperBot') - comments = phbot.getUsersComments(reddit, limitCount=2) + #phbot = get_redditor_by_name(reddit, 'pythonHelperBot') + # comments = phbot.getUsersComments(reddit, limitCount=2) + # Turns out this problem doesn't update the bots submissions, and just + # lives on the submission page + user_name = 'pythonHelperBot' + comments = submission.get_top_level_comments(reddit) already_commented = False + comments_by_bot = [] for comment in comments: # comments link id begins with 't3_' - formatted_Cid = comment.link_id.split('_')[-1] - if formatted_Cid == submission.id: + # formatted_Cid = comment.link_id.split('_')[-1] + # if formatted_Cid == submission.id: + # already_commented = True + if comment.author == user_name: + comments_by_bot.append(comment) already_commented = True + + if already_commented: logging.info("Comment went through despite error message") + if len(comments_by_bot)>1: + logging.error("recorded " + str(len(comments_by_bot) ) + " comments") else: logging.info("After " +str(timeDelay)+ " seconds the comment was not registered") @@ -1195,6 +1209,7 @@ def commentOnSubmmission(submission, msg, reddit, quietMode): if "RATELIMIT" in traceback.format_exc(): logging.error("Caught Server Rate Limit Hit By API | Specific Error:") logging.error("\n"+traceback.format_exc()) + logging.error("Submission ID: " + str(submission.id)) # Bodge already_commented = comment_duplication_by_ratelimit_check(reddit, submission) if already_commented: @@ -1204,6 +1219,10 @@ def commentOnSubmmission(submission, msg, reddit, quietMode): if time.time()-startTime > maxTotalWaitTime: logging.error("I've tried this too much, escalating error") raise e + # Temp Patch: Bodge is failing + if vals_Assigned == False: + logging.warning("Bypassing this block, auto assuming bot has commented, and server is having issues") + vals_Assigned = True else: raise e except (ServerError, ResponseException) as e: From 0361631858235fc3629b4489a05a5e28e1f55748 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Thu, 30 Jan 2020 08:03:44 -0600 Subject: [PATCH 02/10] Added a check for the new help tag --- main.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/main.py b/main.py index 794665f..a8a6874 100644 --- a/main.py +++ b/main.py @@ -135,7 +135,14 @@ def checkForSummons(msg): return summonID - +def check_for_help_tag(submission): + ''' + Checks for new help tag + ''' + if submission.link_flair_text == 'Help': + logging.debug("New Post Tagged as Help") + return True + return False def check_for_key_phrase(submission, phrase_set): botHelperFunctions.logPostFeatures(submission) @@ -151,7 +158,8 @@ def lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set): if key not in oldPosts: submission, user = setOfPosts[key] request_Made = check_for_key_phrase(submission, phrase_set) - if request_Made: + help_tag = check_for_help_tag(submission) + if request_Made or help_tag: submissionsToCommentOn_KP.append(key) return setOfPosts, submissionsToCommentOn_KP From 0ef43c77d885ea33385e5bd57ab16830452301c6 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Thu, 30 Jan 2020 13:35:02 -0600 Subject: [PATCH 03/10] Bot now checks for flair changes and has a protection against users to delete their post before the bot is done grabbing data --- main.py | 41 +++++++++++++++++++++++++++++---- utils/archiveAndUpdateReddit.py | 13 ++++++++--- 2 files changed, 46 insertions(+), 8 deletions(-) diff --git a/main.py b/main.py index a8a6874..157fdfa 100644 --- a/main.py +++ b/main.py @@ -135,15 +135,43 @@ def checkForSummons(msg): return summonID -def check_for_help_tag(submission): +def check_for_help_flair(submission): ''' - Checks for new help tag + Checks for help flair on submissions ''' if submission.link_flair_text == 'Help': - logging.debug("New Post Tagged as Help") + logging.debug("Post Tagged as Help") return True return False +def grab_set_of_submissions_flair(setOfPosts): + ''' + Returns a dictionary of submission id's and associated + flair, making it easy to check if flair has been updated + ''' + flairs = {} + for key in setOfPosts: + submission, user = setOfPosts[key] + flairs[key] = submission.link_flair_text + return flairs + +def check_for_help_flair_update(setOfPosts, old_flairs): + ''' + Returns a list of posts where the help flair has been added + but previously was not on the submission + ''' + submissionsToCommentOn_HF = [] + for key in setOfPosts: + submission, user = setOfPosts[key] + if submission.link_flair_text != old_flairs[key]: + if check_for_help_flair(submission): + logging.debug('Post '+str(key)+' has updated to help flair') + submissionsToCommentOn_HF.append(key) + return submissionsToCommentOn_HF + + + + def check_for_key_phrase(submission, phrase_set): botHelperFunctions.logPostFeatures(submission) request_Made = learningSubmissionClassifiers.request_Key_Word_Classifier(submission, phrase_set) @@ -158,7 +186,7 @@ def lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set): if key not in oldPosts: submission, user = setOfPosts[key] request_Made = check_for_key_phrase(submission, phrase_set) - help_tag = check_for_help_tag(submission) + help_tag = check_for_help_flair(submission) if request_Made or help_tag: submissionsToCommentOn_KP.append(key) @@ -289,8 +317,11 @@ def runBot(reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, # Handle Inbox unreadCount = botSummons.handleInbox(reddit, codeVTextClassifier, phbArcPaths=phbArcPaths, setOfPosts=setOfPosts, unreadCount=unreadCount, sendText= True, quietMode=quietMode) - # Update karma score for posts under 2 hours old + # Update karma score for posts under 2 hours old, and check for help flair + old_flairs = grab_set_of_submissions_flair(setOfPosts) setOfPosts = archiveAndUpdateReddit.updateYoungerThanXPosts(reddit, submissionList=setOfPosts) + submissionsToCommentOn_HF = check_for_help_flair_update(setOfPosts, old_flairs) + commentOnThese += submissionsToCommentOn_HF # Get new posts, respond to keywords setOfPosts, submissionsToCommentOn = lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set) diff --git a/utils/archiveAndUpdateReddit.py b/utils/archiveAndUpdateReddit.py index fc4fb90..3fb26e0 100644 --- a/utils/archiveAndUpdateReddit.py +++ b/utils/archiveAndUpdateReddit.py @@ -1364,9 +1364,16 @@ def getNewPosts(reddit, sub="python", submissionList={}, ageLimitHours=12): #except: # pass time.sleep(1) # Try to reduce rate limit issues - submissionList[submission.id] = [phb_Reddit_Submission(submission)] - user = phb_Reddit_User(reddit.redditor(submission.author.name)) - submissionList[submission.id].append(user) + try: + post = phb_Reddit_Submission(submission) + user = phb_Reddit_User(reddit.redditor(submission.author.name)) + submissionList[submission.id] = [post, user] + except AttributeError as e: + "User Probably deleted the post before the bot got to it | Specific Error:" + logging.error("\n"+traceback.format_exc()) + # submissionList[submission.id] = [phb_Reddit_Submission(submission)] + # user = phb_Reddit_User(reddit.redditor(submission.author.name)) + # submissionList[submission.id].append(user) # ******************************** vals_Assigned = True From 5794a5179c49572237b1525041dc4adbf633afe3 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Wed, 5 Feb 2020 15:43:00 -0600 Subject: [PATCH 04/10] Updating code to 0.4.00 --- main.py | 54 ++++++++++++++++++++------ utils/archiveAndUpdateReddit.py | 6 +++ utils/botHelperFunctions.py | 1 + utils/buildComment.py | 17 +++++--- utils/learningSubmissionClassifiers.py | 20 ++++++---- utils/startupLoggingCharacteristics.py | 28 +++++++++++++ 6 files changed, 102 insertions(+), 24 deletions(-) create mode 100644 utils/startupLoggingCharacteristics.py diff --git a/main.py b/main.py index 157fdfa..e5feb97 100644 --- a/main.py +++ b/main.py @@ -1,3 +1,7 @@ +import sys +from utils import startupLoggingCharacteristics +import logging + import praw import nltk import datetime @@ -5,7 +9,6 @@ import os import argparse, textwrap # Logging Info -import logging import traceback from utils import archiveAndUpdateReddit @@ -35,7 +38,8 @@ ''' def buildHelpfulComment(submission, user, reddit, suggested, crossPosted, answered, - codePresent, correctlyFormatted, quietMode, phbArcPaths): + codePresent, correctlyFormatted, past_interaction, quietMode, + phbArcPaths): supervised = False underDev = True @@ -43,16 +47,38 @@ def buildHelpfulComment(submission, user, reddit, suggested, crossPosted, answer # Intro bagOfSents.append(buildComment.botIntro()) - # User Sittuation Context Awareness + ''' + User Sittuation/Context Awareness: + A few things can happen with a user's question, + They could have asked it in many subs, including r/learnpython + but because they asked it all over the place, so emphasizing the + r/python isn't a place for questions and r/learnpython is is + is important + They could already have their answer, and therefore not need to go to + r/learnpython anymore to get an answer. It still matters that they + know going forward to ask there though, so this takes next priority + Another user could have beat the bot to the comment. It's a good idea + to acknowledge them, but comment anyway to ensure the info in the + comment is relayed to the original poster + Finally, none of the above, so we'll need a catch all introduction + ''' if crossPosted: + # Takes priority bagOfSents.append(buildComment.userCrossPosted()) elif answered: + # Is second because there's no longer a reason to go to r/learnpython bagOfSents.append(buildComment.alreadyAnsweredComment()) elif suggested: + # Next tier because someone beat the bot bagOfSents.append(buildComment.alreadySuggestedComment()) else: + # Finally the catch all of the bot bagOfSents.append(buildComment.standardIntro()) + # Remind them they've seen this before, + if past_interaction: + bagOfSents.append('\n'+buildComment.commented_on_before()) + # Follow rules and help make code clear bagOfSents.append(buildComment.followSubRules()) logging.info("Code Present: " + str(codePresent) + " | Correctly Formatted: " + str(correctlyFormatted)) @@ -242,7 +268,10 @@ def getReadyToComment(reddit, setOfPosts, userNames, postHistory, commentOnThese # Gauge user reactions (right now only archiving) botMetrics.predictUserReaction(reddit, user, phbArcPaths) - buildHelpfulComment(submission, user, reddit, suggested, crossPosted, answered, codePresent, correctlyFormatted, quietMode, phbArcPaths=phbArcPaths) + # Check if allowed to comment even if already commented + past_interaction = user.name in userNames + + buildHelpfulComment(submission, user, reddit, suggested, crossPosted, answered, codePresent, correctlyFormatted, past_interaction, quietMode, phbArcPaths=phbArcPaths) userNames.append(str(user.name)) postHistory.append(str(submission.id)) @@ -332,8 +361,11 @@ def runBot(reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, if datetime.datetime.now() - lastFifteenMin > datetime.timedelta(seconds=fifteenMin*60): #print("15 mins") - # Update posts + # Update posts And Check Flair + old_flairs = grab_set_of_submissions_flair(setOfPosts) setOfPosts = archiveAndUpdateReddit.updatePosts(reddit, submissionList=setOfPosts, phbArcPaths=phbArcPaths) + submissionsToCommentOn_HF = check_for_help_flair_update(setOfPosts, old_flairs) + commentOnThese += submissionsToCommentOn_HF # reclassify posts commentOnThese += handleSetOfSubmissions(reddit, setOfPosts, postHistory, classifier) @@ -389,12 +421,12 @@ def interface(): # Fair assumption that the user is watching the terminal during quiet mode print("Bot is Running in Quite Mode") # Logging Stuff - dirName = "logs" - if not os.path.exists(dirName): - os.makedirs(dirName) - logFileName = 'LOG_'+ datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + '.log' - filePath = os.path.join(dirName, logFileName) - logging.basicConfig(filename=filePath, level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(filename)s:%(funcName)s():%(lineno)s - %(message)s') + # dirName = "logs" + # if not os.path.exists(dirName): + # os.makedirs(dirName) + # logFileName = 'LOG_'+ datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + '.log' + # filePath = os.path.join(dirName, logFileName) + # logging.basicConfig(filename=filePath, level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(filename)s:%(funcName)s():%(lineno)s - %(message)s') if quietMode: logging.debug("Running in Quiet Mode") diff --git a/utils/archiveAndUpdateReddit.py b/utils/archiveAndUpdateReddit.py index 3fb26e0..cf75fc1 100644 --- a/utils/archiveAndUpdateReddit.py +++ b/utils/archiveAndUpdateReddit.py @@ -1160,9 +1160,12 @@ def comment_duplication_by_ratelimit_check(reddit, submission): # formatted_Cid = comment.link_id.split('_')[-1] # if formatted_Cid == submission.id: # already_commented = True + # Under phb_Reddit_Comment, comment.author = praw_comment.author.name + # This prevents accedientally passing around the author class if comment.author == user_name: comments_by_bot.append(comment) already_commented = True + logging.info("Commenters User Names: "+str(comment.author)) if already_commented: @@ -1792,6 +1795,9 @@ def startupDatabase(archive_Locations): def updateDatabase(username, post_id, phbArcPaths): + if phbArcPaths == False: + # Testing + return # Right now it's a flat database, soon it'll be not so flat dirName = "redditData" postCommentedOn = "postHistory.txt" diff --git a/utils/botHelperFunctions.py b/utils/botHelperFunctions.py index a3ae129..10a7428 100644 --- a/utils/botHelperFunctions.py +++ b/utils/botHelperFunctions.py @@ -55,6 +55,7 @@ def logPostFeatures(submission): logging.debug('[POST] | ' + str(submission.title.encode('ascii', 'ignore'))) logging.debug('[AUTHOR] | ' + str(submission.author)) logging.debug('[ID] | ' + str(submission.id)) + logging.debug('[FLAIR] | ' + str(submission.link_flair_text)) postAge = datetime.datetime.utcnow() - submission.created_utc logging.debug( '\t'+"Post Age: "+ str(postAge) ) logging.debug( '\t'+ "Votes: "+ str(submission.score)) diff --git a/utils/buildComment.py b/utils/buildComment.py index 39c7b1e..daa1345 100644 --- a/utils/buildComment.py +++ b/utils/buildComment.py @@ -37,13 +37,18 @@ def userCrossPosted(): r/learnpython, a sub geared towards questions and learning more about python. ''' return msg +def commented_on_before(): + msg = '''I'm sure you've seen this information before, but just in case here it is as a reminder: + ''' + return msg def baseComment(): msg = '''Please follow the subs rules and guidelines when you do post there, it'll help you get better answers faster. Show /r/learnpython the code you have tried and describe where you are stuck. -**[Be sure to format your code for reddit](https://www.reddit.com/r/learnpython/wiki/faq#wiki_how_do_i_format_code.3F)** -and include which version of python and what OS you are using. +If you are getting an error message, include the full block of text it spits out. +**[Here is HOW TO FORMAT YOUR CODE For Reddit](https://www.reddit.com/r/learnpython/wiki/faq#wiki_how_do_i_format_code.3F)** +and be sure to include which version of python and what OS you are using. You can also ask this question in the [Python discord](https://discord.gg/3Abzge7), a large, friendly community focused around the Python programming language, open to those who wish to learn the language @@ -54,12 +59,14 @@ def baseComment(): def followSubRules(): msg = '''Please follow the subs rules and guidelines when you do post there, it'll help you get better answers faster. -Show /r/learnpython the code you have tried and describe where you are stuck. ''' +Show /r/learnpython **the code you have tried and describe in detail where you are stuck.** +If you are getting an error message, include the full block of text it spits out. +Quality answers take time to write out, and many times other users will need to ask clarifying questions. Be patient and help them help you. ''' return msg def formatCodeAndOS(): - msg = '''**[Be sure to format your code for reddit](https://www.reddit.com/r/learnpython/wiki/faq#wiki_how_do_i_format_code.3F)** -and include which version of python and what OS you are using. + msg = '''**[Here is HOW TO FORMAT YOUR CODE For Reddit](https://www.reddit.com/r/learnpython/wiki/faq#wiki_how_do_i_format_code.3F)** +and be sure to include which version of python and what OS you are using. ''' return msg diff --git a/utils/learningSubmissionClassifiers.py b/utils/learningSubmissionClassifiers.py index 8ad6362..1725169 100644 --- a/utils/learningSubmissionClassifiers.py +++ b/utils/learningSubmissionClassifiers.py @@ -179,15 +179,19 @@ def popOldSpammers(antiSpamList, ageLimitHours): # Don't bother commenting if I've talked to the user before + # UNLESS they used the help flair if str(user.name) in userNames: logging.info( "\tI've already commented on a post by "+ str(user.name) ) - msg = "\tI've already commented on a post by " + str(user.name) - print(msg) - if submission.id not in antiSpamList: - antiSpamList[submission.id] = submission.created_utc - msg = msg.strip() + "\n\nPost in Question: "+ botHelperFunctions.shortenRedditURL(submission.url) - textSupervision.send_update(msg) - return False, [], antiSpamList + if submission.link_flair_text != 'Help': + msg = "\tI've already commented on a post by " + str(user.name) + print(msg) + if submission.id not in antiSpamList: + antiSpamList[submission.id] = submission.created_utc + msg = msg.strip() + "\n\nPost in Question: "+ botHelperFunctions.shortenRedditURL(submission.url) + textSupervision.send_update(msg) + return False, [], antiSpamList + else: + logging.info("But they used the help flair") if accountAge > timeDelt: @@ -208,6 +212,6 @@ def popOldSpammers(antiSpamList, ageLimitHours): if directedOthersToLearn: logging.info("User " + str(user.name) + " has directed others to r/learnpython") - return False, [], antiSpamList + #return False, [], antiSpamList return True, postsInLearningSubs, antiSpamList diff --git a/utils/startupLoggingCharacteristics.py b/utils/startupLoggingCharacteristics.py new file mode 100644 index 0000000..91924fa --- /dev/null +++ b/utils/startupLoggingCharacteristics.py @@ -0,0 +1,28 @@ + + +import os +import logging + +import datetime + +dirName = "logs" +if not os.path.exists(dirName): + os.makedirs(dirName) +logFileName = 'LOG_'+ datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + '.log' +filePath = os.path.join(dirName, logFileName) +logging.basicConfig(filename=filePath, level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(filename)s:%(funcName)s():%(lineno)s - %(message)s' ) + +logging.getLogger("praw").setLevel(logging.WARNING) +logging.getLogger("prawcore").setLevel(logging.WARNING) +logging.getLogger("sessions").setLevel(logging.WARNING) +logging.getLogger("rate_limit").setLevel(logging.WARNING) +logging.getLogger("matplotlib").setLevel(logging.WARNING) +logging.getLogger("oauthlib").setLevel(logging.WARNING) +logging.getLogger("requests").setLevel(logging.WARNING) +logging.getLogger("connectionpool").setLevel(logging.WARNING) +logging.getLogger("urllib3").setLevel(logging.WARNING) +logging.getLogger("socket").setLevel(logging.WARNING) +logging.getLogger("requests_oauthlib").setLevel(logging.WARNING) + + +#requests_oauthlib \ No newline at end of file From f6ed5292a7e81aaac8325b49f3e893c2d2363890 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Wed, 5 Feb 2020 15:43:51 -0600 Subject: [PATCH 05/10] Updating documentation to pre alpha A0.4.00 --- CHANGELOG.md | 97 +++++++++++++++- FAQ.md | 16 ++- LICENSE | 2 +- README.md | 14 ++- ROADMAP.md | 321 ++++++++++++++++++++++++++++++++++++++++++++++++++- 5 files changed, 439 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 848f69e..cafba1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ -# CHANGELOG: Python Helper Bot Version pre Alpha A0.3.02 +# CHANGELOG: Python Helper Bot Version pre Alpha A0.3.03 All notable changes to this project will be documented in this file. The format is loosely based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). @@ -18,6 +18,101 @@ Dates follow YYYY-MM-DD format +## [A0.4.00] 2020-XX-XX + +In Progress + +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +This project is not currently looking for other contributors + +#### Big Picture: What happened, what was worked on +The python subreddit has undergone some changes, and now includes flair +on posts. This allows the bot to have a third classifier which it uses to instantly reply on in much the same way the keyword classifier functions. If a post has the 'Help' flair and it was applied within two hours of the original submission, the bot now auto comments on it, presuming it's never commented on that user before. + +If a user has had the bot comment on them before, but uses the help tag, the bot is now allowed to comment again. + +The help message has been adjusted to have all caps on the block which talks about formatting your code. + +Logging has been changed to reduce notes from modules outside of the bot (praw and requests now only shows warnings or higher) + +Some tests have also been added, however it does not cover a majority of the bot yet. + + +#### Added + - check_for_help_flair in main: a function that looks to see if the new flair is being used on the submission and if it's 'Help'. + - grab_set_of_submissions_flair in main: given a set of submissions, returns a dictionary with their id as the key, and their attached flair as the value + - check_for_help_flair_update: as users can put flair on after they post, the bot needs to be able to compare the old flair with the new flair to see if a 'Help' flair has been added. This check currently runs for two hours after a post has been made. + - The bot now should auto reply to help flair added within two hours of the original submission, using the previously listed tools to keep track + - In utils, startupLoggingCharacteristics now exists. It adjusts non-python helper bot logs to warning level, and sets up the structure of the logs thereafter, defining the log structure use to be covered in main. + - Added logging info in archiveAndUpdateReddit's comment_duplication_by_ratelimit_check to make it more useful on review. It's still unclear if this patch solves the issue but this logging should make it easier to notice. +#### Changed + - Updated Year in License + - Added Flair auto-reply explaination in README, de-emphasized bot summons to reflect the lack of active development, added a pre-alpha goal of a reddit post classifier, and adjusted the ethics section to acknowledge the newly allowed rule of commenting on a user multiple times under specific conditions. + - Expanded on FAQ + - Logging should now be more strongly defined by this bot, not sessions and connectionpool, hopefully that'll clean up bug hunts a bit. + - Adjusted the format your code comment to make it more noticable + - In botHelperFunctions logPostFeatures: it now records flair + - In learningSubmissionClassifiers, basicUserClassify now is allowed to comment on a post by a user who had a prior post the bot commented on IF that user uses the 'Help' flair. + - In learningSubmissionClassifiers, basicUserClassify: Bot is now allowed to comment even if user has directed others to learnpython before in the past. + - In botHelperFunctions, added submission_flair_text to the logged post features + - Reworded a lot of the bot's comments in buildComments to emphasize that answers aren't instantanious and added emphasis to the format your code link, since that's the most often ignored aspect of users to follow the bots direction. + +#### Deprecated +#### Removed +#### Fixed + - Under archiveAndUpdateReddit, when evaluating a submission, the bot checks the user of a post and grabs and rewraps it into the wrapper user class. On rare occasions, if a post would be deleted at just the right time, the bot would populate info for the post, the post would be deleted, then the bot would try to populate info for the user which is no longer tied to the now deleted/removed post. A try/except block has been added and now the submissionList waits until both the post and the user info has been successfully built out before it adds it to the list. +#### Security + + +### Main + - The whole logging structure has been updated so that it's now defined in utils/startupLogginCharacteristics. This helps clean up the main and makes it easier to maintain a logging format through the program, preserve and transfer the format to other projects, and suppress uninformative logging messages by imported modules. + - check_for_help_flair, grab_set_of_submissions_flair, and check_for_help_flair_update have been added as have calls to these functions. This collection of functions helps track flair on posts as they've been added as it is not usually added by users right away. The most common reason the bot misses help flair is if the flair was added more than two hours after the initial submission time--reddit rewrites the post age to reflect the age it should be after subtracting the amount of time the submission was removed for. The bot assumes this doesn't happen and may need another change soon to address this issue as well. + - It now checks flair for the full 24 hours it watches a submission. +### rpiManager.py + + +### Util Libraries + +#### archiveAndUpdateReddit.py + - The user issue had been a rare but fatal problem that required a mod to be actively removing a post while the bot was reviewing the same post. Adding the flair bot as a mod increased the chance of this happening, as more posts were removed within 5 minutes of posting if they didn't add flair. I got lucky and was watching the bot when it died and I'm fairly sure this fix will address it, and increase it's resilancy. However the fix logging comment has yet to be saved so it's not absolutely clear that it'll address the issue as a whole. + - In `comment_duplication_by_ratelimit_check`, I've added some comments and a new logging message to increase the information logged when reddit starts breaking and sends a 500 error when it tries to comment. It'll now log all usernames it can see on that post. This should help clairify if the bot's comment is logged by reddit. If reddit is super behind on displaying comments, this error/comment duplication will still occur, but hopefully I'll have more information going forward. +#### botHelperFunctions.py + - Flair is now recorded in logPostFeatures +#### botMetrics.py +#### botSummons.py +#### buildComment.py + - commented_on_before has been added to acknowledge that users have interacted with it before. (This should only activate during help flair--we'll see) + - followSubRules has been adjusted to remind users that people take time to answer questions, and they may not get a reply right away + - formatCodeAndOS has been adjusted to emphasize the link to how to format your code +#### formatBagOfSentences.py +#### formatCode.py +#### learningSubmissionClassifiers.py + - As a result of this version's changes, basic user classify now only prevents comments on users if they've been commented on by the bot before AND are not using the 'Help' Flair. Given this trend, it's reasonable to assume the bot will soon move to comment on users regardless of it's past history with them, as byinlarge the bot is usually correct within a reasonable degree when it's other classifiers activate. +#### locateDB.py +#### lsalib2.py +#### questionIdentifier.py +#### rpiGPIOFunctions.py +#### scriptedReply.py +#### searchStackOverflowWeb.py +#### startupLoggingCharastics.py +#### summarizeText.py +#### textSupervision.py +#### updateLocalSubHistory.py +#### user_agents.py + +### Tests + - Adding comment tree tests: basically purpose different inputs into buildHelpfulComment under main, and ensure the string needed is present + - Added test to make sure test_check_for_help_tag() functions + + ## [A0.3.02] 2019-09-16 Official. diff --git a/FAQ.md b/FAQ.md index a4387a1..5303fea 100644 --- a/FAQ.md +++ b/FAQ.md @@ -11,7 +11,8 @@ If those two basic conditions are met, it'll probably comment. A bit more in depth, - - it'll read the title of the post, if there's a keyphrase present, build a helpful comment. + - It'll read the title of the post, if there's a keyphrase present, build a helpful comment. + - It'll read the flair of the post, and if in the first two hours the post is flaired as 'Help', it'll comment. - If not, then wait a while. - If a post is scoring poorly after a bit, check to see if there's a question in either the title or the body of the submission. - If the submission body is a url that is not the same as the url you'd get if you selected the comments on the thread, (ie if the submission is a 'link post') then the post is ignored. If not, then the post is a 'self post'. @@ -25,7 +26,7 @@ But first it'll look through the top level comments to see if there's already so #### Why didn't the bot comment on this post? -There could be a number of reasons the bot doesn't comment on any given post. +There could be a number of reasons the bot doesn't comment on any given post. In fact, there's probably more reasons the bot doesn't comment than reasons the bot comments. The post could be performing well enough that the bot classifies the post as a 'possibly interesting question'. What "is" and "isn't" interesting is not really for the bot to decide, so it looks to the post karma to help make that decision. @@ -34,6 +35,8 @@ The post could be written in a way that the classifier is unable to classify a s The bot keeps a list of users who's posts it has commented on. To prevent spam the bot doesn't comment on posts made by those users in the future. (This is under the assumption that once someone is aware of the r/learnpython, they'll use it in the future. This assumption has proven to be rather strongly false and I'm reevaluating this action, but for now I'm maintaining this behavior.) +If a user has already posted in r/learnpython by the time the bot decides to comment, but the post wasn't made at about the same time as the original, the bot decides its comment isn't needed. + #### The bot shouldn't have commented @@ -43,6 +46,10 @@ There are also key phrases that auto-trigger the bot, because they're ngrams whi Acknowledging these issues, the bot is set to have fewer false positives than false negatives, so hopefully this isn't a frequent issue. +If the bot really shouldn't have commented, downvote it. This gives the bot a metric that's more useful than 'good bot'/'bad bot' comments as more users vote than comment. + +#### I'm not learning python, I'm asking an advanced question +r/learnpython isn't a subreddit for beginners. While it might be specifically geared so that questions common to new programmers are easily addressed, the sub functions as a sister sub to r/python. Together one handles most news based submissions, and the other handles most questions. There is of course some overlap, but being a capable python programmer does not prevent anyone from asking a question and getting an answer on r/learnpython. #### Shouldn't the bot Direct Message the OP rather than commenting? I thought about private messaging, but then you'll have multiple people (or bots) performing the same action without anyone being able to see. From the perspective of anyone on the receiving end, this could be a massive amount of spam. @@ -67,6 +74,9 @@ If you purified it down to its base element, carbon, you might be able to make s The most important thing to consider here is the fact that I have no idea what I'm talking about, and shouldn't be listened to on the topic of material engineering. +#### Why comment when someone else already directed the user to r/learnpython +The bot decides to acknowledge the user but still comment in this case because the bot has a scripted 'help' note that emphasizes how to ask the question, and includes helpful links to help format the code. Its important to make sure new programmers understand how to ask a question and what information they need to help others help them. + #### The question was already answered, why did the bot still comment The bot has a hard time telling whether or not an answer was already given, and if it was useful. There's code in the works to try to see if the OP has said "Thank you" or "That worked" or "It works now" but it's hard to test and not very reliable. @@ -124,4 +134,4 @@ When I get to that point, I'll probably just have folks tackle specific elements They seem cool. I've got no problem with them. -#### Version Pre Alpha A0.3.02 \ No newline at end of file +#### Version Pre Alpha A0.4.00 \ No newline at end of file diff --git a/LICENSE b/LICENSE index abb75bf..5e52577 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright 2018-2019, Keith Murray +Copyright 2018-2020, Keith Murray Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/README.md b/README.md index 05d08b9..029d30d 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# Reddit Helper Bot: Version Pre Alpha A0.3.02 +# Reddit Helper Bot: Version Pre Alpha A0.4.00 pythonHelperBot is a reddit bot built to analyze r/python post and determine if they're better suited for the r/learnpython sub. If they are it suggests that the user post to that sub rather than to r/python. @@ -15,6 +15,7 @@ If those two basic conditions are met, it'll probably comment. A bit more in depth, - it'll read the title of the post, if there's a keyphrase present, build a helpful comment. + - It'll read the flair of the post, and if in the first two hours the post is flaired as 'Help', it'll comment. - If not, then wait a while. - If a post is scoring poorly after a bit, check to see if there's a question in either the title or the body of the submission. - If the submission body is a url that is not the same as the url you'd get if you selected the comments on the thread, (ie if the submission is a 'link post') then the post is ignored. If not, then the post is a 'self post'. @@ -27,6 +28,8 @@ But first it'll look through the top level comments to see if there's already so ## Summoning The Bot +While the bot can be summoned for some small tasks, it really isn't a feature that is being used nor is it a feature that I feel offers much to the end user. + ### Commands: - `/u/pythonHelperBot !reformat` - `/u/pythonHelperBot !format_howto` (not yet active) @@ -38,7 +41,7 @@ The bot is only allowed to make three reformatting comments on the same submissi #### `/u/pythonHelperBot !format_howto` -Will Be Active Soon +Currently inactive. The bot can also be summoned to display a helpful message about how to format code for reddit. This message will appear below the summoning comment, and tag the part post/comment in its message. Because of this, it tags a user unprompted, it's only allowed to interact with that user once, unless the summoner and parent post/comment are by the same user. @@ -53,6 +56,7 @@ The bot is currently in a pre alpha stage. This means that the founding goals of - [X] Simply comment on redditors posts who look like they should post in r/learnpython - [X] Run on the raspberry pi - [X] Archive reddit posts for future classification evaluation + - Build a reddit post classifier using the archive and LDA - Build a local Stack Overflow Search Engine - Use Stack Overflow to gauge the simplicity of a redditors question - Use Stack Overflow to implement a naive Question and Answer system @@ -98,10 +102,10 @@ This bot is intended to make a healthy recommendation that the user go to the ap Therefore, this bot should not be spammy. -This is enforced by never allowing the bot to suggest r/learnpython to a user more than once, as it is assumed that after that point the user should be aware of the sub. -This assumption is known to be invalid because I've been on the internet before, and [people really are just the worst](https://www.youtube.com/watch?v=m0KFY6o6unw) ([further](https://www.youtube.com/watch?v=fZv_TARX3lI)). +This is enforced by almost never allowing the bot to suggest r/learnpython to a user more than once, as it is assumed that after that point the user should be aware of the sub. +This assumption is known to be invalid because I've been on the internet before, and [people really are just the worst](https://www.youtube.com/watch?v=m0KFY6o6unw) ([further](https://www.youtube.com/watch?v=fZv_TARX3lI)). The currently allowed exception to this is if they flair their submission as 'Help', as that is a self applied classification and the bot can be confident in it's actions. -Unless a key phrase is used in the title of the post, the bot is not allowed to comment on a post until a certain amount of time has passed, which allows votes to be added to the feature vector as a secondary classification step. +Unless a key phrase is used in the title of the post, or the post has 'Help' flair, the bot is not allowed to comment on a post until a certain amount of time has passed, which allows votes to be added to the feature vector as a secondary classification step. The bot is not currently allowed to remove a comment after it's clearly made a mistake. This is typically defined as having a comment score of less than -3 and the post having a karma score of greater than 1 diff --git a/ROADMAP.md b/ROADMAP.md index 0940cc4..53394a9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -23,8 +23,327 @@ Dates follow YYYY-MM-DD format -## [A0.3.02] 2019-09-16 +## [A0.3.03] 2020-XX-XX + In Progress + +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +### Short Term Roadmap + +Address Flair and being adding tests. When min_spanning_tree program is complete, as well as when 'fix reddit archive' is done, begin rolling out a new classifier using LDA + +#### Add + - [X] Migrate most files to usb/usb-sata drive to host. Watch power requirements. + - With light archive, build a 'save state' and 'load state' function so records on new posts aren't lost during reboots. + - [X] loggingSetup.py: a module to be imported first by rpiManager and main, which sets up the logging format so the program can be called by either module on any system and initialize in the same way + - botHelperFunctions/botMetrics: Add ram usage check and log it in every 'awake' cycle of program, to try and tease out any possible MemeoryError's kicking up after long periods of time. Might as well save other diagnostics info, maybe call this from the hearbeat thread so it becomes a better representation of runtime + - [X] rpiManager.startupSwitchFlag(): IGNORED + Also a pull origin master from github state would be incredibly useful + - [X] Verify reddit post logs by grabbing most recent bot comments. This should reduce risk of two computers commenting on the same post which could happen if databases are de-synced and one computer is not in quiet mode because I typed in the wrong command. Opperator Error Risk Reduction. Only the most recent x+buffer hours of interaction are needed. This does not protect from multiple posts by the same user but should prevent multiple comments on the same post even if the bot runs on different computers. + - Full Test Suite: PRIORITY (Ok it'll have to wait until posts have been archived, but it'll still be nice) + - [X] Archive Posts: PRIORITY + Solidify what values to save, and what to save them with. Probably build an SQL to XML or JSON exporter for third party testing. + - botMetrics.measureUserReaction(): + A function focused on seeing if a user did in fact go to + r/learnpython after the bot made its suggestion. Currently built (kind of, the praw wrappers messed it up a bit), now need to add + functionality in main.py to use it + - Continue Documentation in functions, add documentation files too + - A queue which has inputs added to it by functions, and removed by the raspberry pi gpio handler, allowing the bot to change LED status based on what it's doing. Similar to the twitter event bot design (sepperate project). + - Deprecate karma scatter plot: or change it post all posts in the past week to minimize size. It's no longer useful or very interesting. Maybe activate it once a week or something too + - [X] Consider creating praw rewrapper (prawRegift) to hold all praw focused wrappers and sepperate it from the phb functions. This will make it easier to have the same protections on other bots as necessary. + - botMetrics.predictUserReaction(): A function to go through users comment history, look at the parent comments, and from that gauge how the user will respond to the bots help. In the future adjust how the bot replies based on the predicted responsiveness. For now, it'll just build an archive of users responses to previous comments. + + + +#### Change + - Move NLP functions from NLTK (used in many files) to a buffer module, allowing for simpler, universal changes to be made. For example, if there is a better POS tagger than nltk.pos_tag(sent) then we can easily switch to that. Right now NLTK is used over a fairly large filespace making adjustments of that sort difficult. + - Review all my logging notes. See what should be dropped, changed, etc. + - Make sure the bot defaults to commenting about formatting even if there's no code present + - When errors occur, or a shutdown button is pressed, attempt to save current reddit info into a temp file. on startup, load that info in then connect with praw to expand the knowledge base. + - The archive and update reddit module has become unwieldy. It should be broken into two modules, one which handles the recasting of the praw classes, and one which deals with them as needed. + - Migrate away from usage of my personal libraries so it's easier for others to get the bot up and running. + + +#### Deprecate + +#### Remove + +#### Fix + - Standardize function name style. Either underscore or camelcase, just not both + Probably preferable to use underscore, the despite camelcase being faster.. + +#### Security +#### Consider + + + +--- + +### General to Long Term Expansion + + - Develop terms for a walk away condition. Either End of active development and the bot remains online, end of active development and death of bot, or end of active development and project is passed on to others. Terms will almost certainly be changed constantly and the project grows and evolves, but it's nice to have an idea of what I consider to be a "complete" project. + + - Numbering system for items in roadmap to clear up what's being worked on and what is completed from an outside perspective. A master numbering system probably is a good idea, vX.X.XX[a,c,d,r,f,s,co]XX, following version, section, and specific roadmap suggestion number. But That seems bloated and unnecessary. (Maybe this isn't worth while, maybe it is and will help catch things in the changelog. Probably wont be seriously considered until alpha) + + - [X] summarizeText.loadEnglishModel(sourceDataPath=paths["prebuiltEnglishDB"]): + Prebuild and pickle the output tdm of + summarizeText.buildModelFromDocsInFolder(sourceDataPath=paths["englishDB"]) + so the raspberry pi doesn't have to hit memory errors in in main.startupBot() + load the prebuilt database and build it if the prebuild database does not exist. + Or just build a custom compression scheme and load that instead of using pickle. + + - test.EvaluatePost(): + Given recent restructuring, this should be much easier. Take a post given a post id, then run it through the classifier where the exitpoints are turned off from the functions, forcing it to classify the post in full. Because the praw wrappers are in place, there shouldn't be a concern about forcing full evaluation any more. This can be considered to be half completed: the silent mode the bot has helps evaluate posts. + + - Reply To Common posts: + Build semi scripted replies to frequently asked questions (probably largely pulled from the sidebar, since that's how the side bar gets populated) + This includes "How do I install python" and "Is learning python worth it?" (Aka "Why learn python?")--A quality version of this is a full research task. Build off of internal sentence ordering models + soft skills. Pull text from comments on these posts to build up scripted reply. Look into adapting this (or maybe starting with this) to classifying the posts so their types can be diplayed as tags. + + - local Flask website/dashboard to monitor the status and logs of the bot in realtime. Low priority. + + - Log processing: a set of functions and visualizations to process the log files for various useful tidbits. Something nicer than grep + + + + +### Main + - alreadyAnswered(): + Parse through OPs comments on the thread, and search for text that implies the question + has been answered. Adjust comment on submission accordingly, probably to say, "Next time + you have a question like this, consider using r/learnpython" blah blah blah + + +### rpiManager.py + - update the commented gpio naming and numbering list + - update grab-from-github functions + - add a queue to work with rpiGPIO for LED displays for various tasks + + + +### Libraries: + + +### archiveAndUpdateReddit.py +Most of the 'light' archive functionallity is currently being built out, rendering a large chunk of this section of the roadmap either completed, in progress, or dismissed. It is also no longer in the realm of 'long term'. + +[x] The big set of functions necessary in this module are database creation, and update functions. +There might be two databases: one of just posts, and another comprising of posts, and comments. + +ARCHIVE FUNCTIONS +TODO: Saves it into sqlite3 table after it's passed the time threshold to "not in +use" + +"not in use" is probably going to be defined as 8 hours. Past that point the +post will be either 'successful', 'mild', or 'unsuccessful', defined as +x >= 8 points, 8 > x >= 1, 1 > x + +Adjust it so 'not in use' is not defined as 8 hours, but instead defined as +a varible, which changes based on the time of day (either defined by utc or +cdt--cdt being my current local time) that the posts was made. 'Late at night' as +defined by the time where the fewest users/ r/python 'actions' (posts, upvotes, +comments) are made, adjusted according to the day of the week and or holiday +(unlikely that this bot will need to be that specific) the post is made on. + + r/learnpython is another source of data: + +it will act as a source of useful questions, and will allow the bot to direct +users to other reddit based questions rather than simply stack overflow (this +distinction should help allow the bot to be generalizable) +posts between 3 and 8 upvotes will be determined as 'basic questions' and +will be used as suggested solutions if the similarity between the new r/python +post and the old r/learpython post is greater than some threshold + + +### botHelperFunctions.py + +### botMetrics.py + - predictUserReaction(): used for the bot gauge how receptive the user will be towards a r/learnpython suggestion. If it predicts combative, it'll make it's comment short and to the point. If it predicts receptive it'll expand on highlighted points such as formatting. If neither it'll contiue as is. + + - measureUserReaction(): +to see if redditor does post to r/learnpython. The post will have to be strongly similar +to their r/python post, and be posted not long after the python sub post + +- questionAndAnswer(query): to attempt to reply to semi-scripted questions + +- buildConfusionMatrix(): to measure performance + +#### Confusion Matrix Traits +##### True Positive: +[The bot has commented,] And +[[Either a mod has removed the post due to 'learning'], +Or [the redditor posts their question on r/learnpython]] + +##### False Negative: +[The bot did not comment after 8 hours] And +[[Either a mod has removed the post due to learning,] +Or, [[someone else has commented r/learnpython] and [has greater than 2 upvotes after 8 to 24 +hours after commenting,]] +Or, [the user posted their question on r/learnpython]] + +##### False Positive: +[The bot has commented],And [has less than -1 comment karma after 8 to 24 hours,] +And [[the post is not removed due to learning within 8-24 hours] or [by mods recent +activity plus some threshold.]] +And [[the user does not make a similar post to r/learnpython within a timespan of 8 +hours] or [4 hours after their next user activity monitored for no more than a week]] + +##### True Negative: +[The bot did not comment after 8 hours] And +[[No mod has removed the post using a reference to 'learning' after 8 to 24 hours] or +by mods recent activity plus some threshold.] And +[[No commenter has post made a post which contains 'r/learnpython'] And [has more than 2 upvotes]] + +##### Fuzzy: +All Else. +This class will be either require human moderation to place into the confusion matrix, Or +will be used for other classificaiton, such as "Blog Spam". It could be that topics placed +in this area can be re-examined and labelled, helping the bot generalize preformance in +other areas + +### botSummons.py + - Finish makeFormatHelpMessage summons +### buildComment.py +### formatBagOfSentences.py +### formatCode.py + - formatCode.py: Cleave sentence from comment and first line of code from one another + - formatCode.py: Using rewrapClassifications output, check to see if any indentation is present for lines that have been classified as code. If >5 lines of code are present and none of them have indents, classify block as "The reddit text editor royally screwed this one up", adjust comment to say it's unlikely that the code has been indented properly, and enter the special fixer. + - formatCode.reformatFromHell(): Read in all previous code. Read in current line. If rfh classification Adds indent: current line is a child of the previous line. If it is the same indent level, current line is a sibling. If it is minus indent, line is a sibling of the previous lines parent. + - Previous code is stored in a tree like structure + - Leverage sentence ordering ideology to say given the current line and the previous state of the code tree, which level of node in the tree should I be + This should be an area of linguists where there's plenty of work already completed, look for it. I think Nevil-manning sequitor addresses it briefly, look at that+cited by for other work in the area. + + +### learningSubmissionClassifiers.py + +### locateDB.py + - load in path data from a prefernce file, and or take it as input that way the path isn't + 1. Hard coded and + 2. Hard coded in the module + Generic is better if it's generally useful. + + That said, "check_though_these():" is a pretty good and simple function to move out of "locateDB.py" and into main.py + + - Call a function in this library to recast folder/file calls to the correct os format. Or just redo it everywhere in the code. + Whatever works best + +### lsalib2.py + +### questionIdentifier.py +It'd be nice to use stack overflow's user submissions and r/learnpython's +submissions compared to 'successful' r/python submissions to build a 'programmers +question' classifier (and expand the classifier to blogspammers). This would +make it generalizable so posts which are questions or requests ("HELP ME CODE") +are directed to r/learnpython, posts which are clearly for click/ads are commented +on as such, and good posts are 'ignored': allowing redditors to act on it as they +choose. This is not an easy goal to acheive and is incredibly arbitary. but there +are still certain factors which can be measured and acted on. + +This will probably leverage a stack overflow search engine and compare n results +with k or greater similarity. + +### rpiGPIOFunctions.py +### scriptedReply.py +### searchStackOverflowWeb.py + - Scrap and rebuild with approved api and bound it to search for results between + local database build date and present day. Not important until after local copy of SO + is up and running +### summarizeText.py + - Improve the english language model for topic modeling, and focus on programming topic modeling. +### textSupervision.py +### updateLocalSubHistory.py +### user_agents.py + +### OTHER +(This is all functions that don't have a clear parent module) + + - moqaProgram + + - ELMO/BERT programs + + - reformat_User_Code(): + a function to identify python code blocks that aren't properly formated, and auto format the code + for other reddit users. Might live it its own module. + Currently being worked on. + + - Leverage reformat user code with automatic Q&A: Use classified code regions to match SO code regions, classified text regions to match SO text regions. Hopefully this improves the search engine and cuts the risk of added noise by a text to code block increasing precieved distance between the user query and the SO database post. + Next If a majority of highly matching SO posts have sample code in the question, but the reddit query does not, strongly suggest adding the example code that caused the issue to the next itteration of the query. + + + - question_topic_Modeling(): + This is going to take a few parts. + - Identify all related learning subreddits: + + - Model the topics of stack overflow questions. + + - Model the topics in the learning subs + Do network analysis to find the most active sub that addresses a topic: probably pagerank since it's simple and it works. It doesn't need to be state of the art, and if it can run on the pi, that's even better + + - Next take in the question, extract topics, feed the topics in the network, identify the sub that will get the best answer fastest. This means there also has to be some knowledge of the subs activity score + + - sub_Activity_Measure(): + Or score.. + This will probably return some arbatrary number that only makes sense in the context of other measures + It might be a function of: + The distance between the top 25 posts on Hot and the top 25 posts in New, where 'top' refers to + reddits ranking. + The number of comments and the absolute value of karma of those comments + the number of unique users in those 25 posts + The time between each activity + + Comparing the intersection of hot to new posts shows a glimps of how active the sub is without requiring the bot to look at the sub at multiple times. + + This function would be useful with the question_topic_modeling() function and wouldn't need to run frequently. Though over multiple runs, it would have a solid understanding of how active a sub is at different times of day, which might encourage the bot to direct a user to a learning sub that is + active at that time. + + + - Auto Reply to common questions (Functional FAQ as it were) + (This is probably going to be an early test of soft skills) + * ["Possibly wanting to learn Python, is it worth it?"](https://www.reddit.com/r/Python/comments/917zxd/) + + - Use Automatic Sentence Ordering to construct the bots autoreply, reducing the mess of the code there. Should be mildly simple (ha, sure...), and allow for much more flexible commenting. Target is to have a defined intro, a 'bag of sentences' for the body, and a defined signature. The 'mildly simple' notion is built off the idea that there will be little the program can do incorrectly with that scaffolding. Look at two metrics: absolute sentence ordering, and new paragraph insertion. Maybe train on a ton of readme's, or wiki data for the new paragraph insertion. + + +#### Question & Answer +Resources to draw from: + - Stack Overflow (Primary) + - Python Docs (Secondary) + - Python Blog Posts (Out of Focus) + - Scraped Github Code (Out of Focus) + +Think about using a subset of highly matching SO posts code to OPs source code and using bayes in a MSAlignment fashion to guess on solution. +Most likely this is especially useful with syntax errors and stack traces. + + +### Generalizing the bot: +These are features which an ideal bot-mod would have, but which are not directly linked to a question-answer-and-redirector bot like u/pythonHelperBot (as of mid July 2018) + - sub_Toxicisty_Score(): + Alternatively a friendly score. Bit ambigous, and doesn't immeadetly fit into the bot, but just a measure of how kind or standoffish or toxic a sub is. Certain communities tend to forget that not everyone knows everything, and it'd be nice to avoid recommending those subs. + + - blog_Spam_Flagger(): + This is actually a large but distant future goal for the bot. There's often complaints about blog spam on the python sub, and it'd be nice to have a programmatic way to define it. Even if the spammy site sees the definition, and works around it, the definition can either be altered, or the work around can be allowed. Most redditors want to see good content, so the best way around an ideal blog spam filter would be to have variable, high quality content. In which case everyone wins. Using that idea, we can start to outline the basic components of what blog spam might be. + + High quality content is safe. High quality with respect to the python sub is probably some function of what generally does well + + Low quality can be caused by a few reasons: r/python is not the proper sub for that: ie questions + It was recently posted: this is probably best defined as content theft, though repost is a common name for it. + + I'm tired, I'll come back to this. + +### Tests + + +## [A0.3.02] 2019-09-16 + +Official. + ### Contributors Keith Murray From 20792de4e1886907bcf7cf991cd6cb9f4235baaf Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Fri, 14 Feb 2020 06:19:09 -0600 Subject: [PATCH 06/10] Updating documentation and bot help messages --- CHANGELOG.md | 4 ++- README.md | 3 ++ ROADMAP.md | 73 ++++++++++--------------------------------- rpiManager.py | 17 +++++----- utils/botMetrics.py | 2 +- utils/botSummons.py | 3 ++ utils/buildComment.py | 8 ++--- 7 files changed, 39 insertions(+), 71 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cafba1d..233d188 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ -# CHANGELOG: Python Helper Bot Version pre Alpha A0.3.03 +# CHANGELOG: Python Helper Bot Version pre Alpha A0.4.00 All notable changes to this project will be documented in this file. The format is loosely based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). @@ -64,8 +64,10 @@ Some tests have also been added, however it does not cover a majority of the bot - In learningSubmissionClassifiers, basicUserClassify: Bot is now allowed to comment even if user has directed others to learnpython before in the past. - In botHelperFunctions, added submission_flair_text to the logged post features - Reworded a lot of the bot's comments in buildComments to emphasize that answers aren't instantanious and added emphasis to the format your code link, since that's the most often ignored aspect of users to follow the bots direction. + - Added a comment that r/learnpython is the place for questions regardless of how advanced the question is. #### Deprecated + - Commented out the call to karmaPlot in botMetrics. It'll need to be more fully removed later on, but the bot no longer needs it. #### Removed #### Fixed - Under archiveAndUpdateReddit, when evaluating a submission, the bot checks the user of a post and grabs and rewraps it into the wrapper user class. On rare occasions, if a post would be deleted at just the right time, the bot would populate info for the post, the post would be deleted, then the bot would try to populate info for the user which is no longer tied to the now deleted/removed post. A try/except block has been added and now the submissionList waits until both the post and the user info has been successfully built out before it adds it to the list. diff --git a/README.md b/README.md index 029d30d..9930841 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,9 @@ If they are it suggests that the user post to that sub rather than to r/python. - [FAQ](https://github.com/CrakeNotSnowman/redditPythonHelper/blob/master/FAQ.md) +The current incarnation of the bot is aimed at encouraging the use of the r/learnpython sub. +It may flag all learning posts, but it only comments if it (in a naive way) determines that the user does not know about or chooses not to use the learning sub. + ## What is it doing? diff --git a/ROADMAP.md b/ROADMAP.md index 53394a9..21d4a9e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ -# ROADMAP: Python Helper Bot Version pre Alpha A0.3.02 +# ROADMAP: Python Helper Bot Version pre Alpha A0.4.00 Future expansions are considered in this file. Their presence is not a promise that they'll exist, but rather this file serves as an early outline of features this project hopes to add, as well as changes in directions @@ -23,7 +23,7 @@ Dates follow YYYY-MM-DD format -## [A0.3.03] 2020-XX-XX +## [A0.4.00] 2020-XX-XX In Progress @@ -38,27 +38,21 @@ Unless otherwise noted, all changes by @kmurrayis ### Short Term Roadmap -Address Flair and being adding tests. When min_spanning_tree program is complete, as well as when 'fix reddit archive' is done, begin rolling out a new classifier using LDA +Address Flair and begin adding tests. When min_spanning_tree program is complete, as well as when 'fix reddit archive' is done, begin rolling out a new classifier using LDA #### Add - - [X] Migrate most files to usb/usb-sata drive to host. Watch power requirements. - With light archive, build a 'save state' and 'load state' function so records on new posts aren't lost during reboots. - - [X] loggingSetup.py: a module to be imported first by rpiManager and main, which sets up the logging format so the program can be called by either module on any system and initialize in the same way - botHelperFunctions/botMetrics: Add ram usage check and log it in every 'awake' cycle of program, to try and tease out any possible MemeoryError's kicking up after long periods of time. Might as well save other diagnostics info, maybe call this from the hearbeat thread so it becomes a better representation of runtime - - [X] rpiManager.startupSwitchFlag(): IGNORED Also a pull origin master from github state would be incredibly useful - [X] Verify reddit post logs by grabbing most recent bot comments. This should reduce risk of two computers commenting on the same post which could happen if databases are de-synced and one computer is not in quiet mode because I typed in the wrong command. Opperator Error Risk Reduction. Only the most recent x+buffer hours of interaction are needed. This does not protect from multiple posts by the same user but should prevent multiple comments on the same post even if the bot runs on different computers. - Full Test Suite: PRIORITY (Ok it'll have to wait until posts have been archived, but it'll still be nice) - - [X] Archive Posts: PRIORITY - Solidify what values to save, and what to save them with. Probably build an SQL to XML or JSON exporter for third party testing. - botMetrics.measureUserReaction(): A function focused on seeing if a user did in fact go to r/learnpython after the bot made its suggestion. Currently built (kind of, the praw wrappers messed it up a bit), now need to add functionality in main.py to use it - Continue Documentation in functions, add documentation files too - - A queue which has inputs added to it by functions, and removed by the raspberry pi gpio handler, allowing the bot to change LED status based on what it's doing. Similar to the twitter event bot design (sepperate project). - - Deprecate karma scatter plot: or change it post all posts in the past week to minimize size. It's no longer useful or very interesting. Maybe activate it once a week or something too - - [X] Consider creating praw rewrapper (prawRegift) to hold all praw focused wrappers and sepperate it from the phb functions. This will make it easier to have the same protections on other bots as necessary. + - LED Status: A queue which has inputs added to it by functions, and removed by the raspberry pi gpio handler, allowing the bot to change LED status based on what it's doing. Similar to the twitter event bot design (sepperate project). + - [X] Deprecate karma scatter plot: or change it post all posts in the past week to minimize size. It's no longer useful or very interesting. Maybe activate it once a week or something too - botMetrics.predictUserReaction(): A function to go through users comment history, look at the parent comments, and from that gauge how the user will respond to the bots help. In the future adjust how the bot replies based on the predicted responsiveness. For now, it'll just build an archive of users responses to previous comments. @@ -66,9 +60,10 @@ Address Flair and being adding tests. When min_spanning_tree program is complete #### Change - Move NLP functions from NLTK (used in many files) to a buffer module, allowing for simpler, universal changes to be made. For example, if there is a better POS tagger than nltk.pos_tag(sent) then we can easily switch to that. Right now NLTK is used over a fairly large filespace making adjustments of that sort difficult. - Review all my logging notes. See what should be dropped, changed, etc. - - Make sure the bot defaults to commenting about formatting even if there's no code present + - [X] Make sure the bot defaults to commenting about formatting even if there's no code present--It just thinks there's code a touch too often + - If it classifies a line as code, try to validate the expression in a basic form. This might improve the classifier and reduce isThereCode False Positives. - When errors occur, or a shutdown button is pressed, attempt to save current reddit info into a temp file. on startup, load that info in then connect with praw to expand the knowledge base. - - The archive and update reddit module has become unwieldy. It should be broken into two modules, one which handles the recasting of the praw classes, and one which deals with them as needed. + - The archive and update reddit module has become unwieldy. It should be broken into two modules, one which handles the recasting of the praw classes, and one which deals with them as needed.--Ehh.. Maybe not. It does need to be together to a degree - Migrate away from usage of my personal libraries so it's easier for others to get the bot up and running. @@ -93,19 +88,13 @@ Address Flair and being adding tests. When min_spanning_tree program is complete - Numbering system for items in roadmap to clear up what's being worked on and what is completed from an outside perspective. A master numbering system probably is a good idea, vX.X.XX[a,c,d,r,f,s,co]XX, following version, section, and specific roadmap suggestion number. But That seems bloated and unnecessary. (Maybe this isn't worth while, maybe it is and will help catch things in the changelog. Probably wont be seriously considered until alpha) - - [X] summarizeText.loadEnglishModel(sourceDataPath=paths["prebuiltEnglishDB"]): - Prebuild and pickle the output tdm of - summarizeText.buildModelFromDocsInFolder(sourceDataPath=paths["englishDB"]) - so the raspberry pi doesn't have to hit memory errors in in main.startupBot() - load the prebuilt database and build it if the prebuild database does not exist. - Or just build a custom compression scheme and load that instead of using pickle. - - test.EvaluatePost(): Given recent restructuring, this should be much easier. Take a post given a post id, then run it through the classifier where the exitpoints are turned off from the functions, forcing it to classify the post in full. Because the praw wrappers are in place, there shouldn't be a concern about forcing full evaluation any more. This can be considered to be half completed: the silent mode the bot has helps evaluate posts. - Reply To Common posts: Build semi scripted replies to frequently asked questions (probably largely pulled from the sidebar, since that's how the side bar gets populated) This includes "How do I install python" and "Is learning python worth it?" (Aka "Why learn python?")--A quality version of this is a full research task. Build off of internal sentence ordering models + soft skills. Pull text from comments on these posts to build up scripted reply. Look into adapting this (or maybe starting with this) to classifying the posts so their types can be diplayed as tags. + - When LDA classifier comes into play, this might be a lot easier. Auto-segment posts into topics, ID topic of question, compare question to similar past question, map past answers to new question. - local Flask website/dashboard to monitor the status and logs of the bot in realtime. Low priority. @@ -122,7 +111,7 @@ Address Flair and being adding tests. When min_spanning_tree program is complete ### rpiManager.py - - update the commented gpio naming and numbering list + - update the commented gpio naming and numbering list at the top of the file - update grab-from-github functions - add a queue to work with rpiGPIO for LED displays for various tasks @@ -132,40 +121,12 @@ Address Flair and being adding tests. When min_spanning_tree program is complete ### archiveAndUpdateReddit.py -Most of the 'light' archive functionallity is currently being built out, rendering a large chunk of this section of the roadmap either completed, in progress, or dismissed. It is also no longer in the realm of 'long term'. - -[x] The big set of functions necessary in this module are database creation, and update functions. -There might be two databases: one of just posts, and another comprising of posts, and comments. - -ARCHIVE FUNCTIONS -TODO: Saves it into sqlite3 table after it's passed the time threshold to "not in -use" - -"not in use" is probably going to be defined as 8 hours. Past that point the -post will be either 'successful', 'mild', or 'unsuccessful', defined as -x >= 8 points, 8 > x >= 1, 1 > x - -Adjust it so 'not in use' is not defined as 8 hours, but instead defined as -a varible, which changes based on the time of day (either defined by utc or -cdt--cdt being my current local time) that the posts was made. 'Late at night' as -defined by the time where the fewest users/ r/python 'actions' (posts, upvotes, -comments) are made, adjusted according to the day of the week and or holiday -(unlikely that this bot will need to be that specific) the post is made on. - - r/learnpython is another source of data: - -it will act as a source of useful questions, and will allow the bot to direct -users to other reddit based questions rather than simply stack overflow (this -distinction should help allow the bot to be generalizable) -posts between 3 and 8 upvotes will be determined as 'basic questions' and -will be used as suggested solutions if the similarity between the new r/python -post and the old r/learpython post is greater than some threshold ### botHelperFunctions.py ### botMetrics.py - - predictUserReaction(): used for the bot gauge how receptive the user will be towards a r/learnpython suggestion. If it predicts combative, it'll make it's comment short and to the point. If it predicts receptive it'll expand on highlighted points such as formatting. If neither it'll contiue as is. + - predictUserReaction(): used for the bot gauge how receptive the user will be towards a r/learnpython suggestion. If it predicts combative, it'll make it's comment short and to the point. If it predicts receptive it'll expand on highlighted points such as formatting. If neither it'll contiue as is. --There's probably almost no meat to go off of to help make this prediction though, it's most likely a good idea with no path to implementation. - measureUserReaction(): to see if redditor does post to r/learnpython. The post will have to be strongly similar @@ -179,7 +140,7 @@ to their r/python post, and be posted not long after the python sub post ##### True Positive: [The bot has commented,] And [[Either a mod has removed the post due to 'learning'], -Or [the redditor posts their question on r/learnpython]] +Or [the redditor posts their question on r/learnpython], Or [the bot has >=2 upvotes]] ##### False Negative: [The bot did not comment after 8 hours] And @@ -235,6 +196,7 @@ other areas Whatever works best ### lsalib2.py +Migrate features back into lsalib ### questionIdentifier.py It'd be nice to use stack overflow's user submissions and r/learnpython's @@ -255,11 +217,13 @@ with k or greater similarity. - Scrap and rebuild with approved api and bound it to search for results between local database build date and present day. Not important until after local copy of SO is up and running +### startupLoggingCharastics.py ### summarizeText.py - Improve the english language model for topic modeling, and focus on programming topic modeling. ### textSupervision.py ### updateLocalSubHistory.py ### user_agents.py + - Remove this ### OTHER (This is all functions that don't have a clear parent module) @@ -268,18 +232,13 @@ with k or greater similarity. - ELMO/BERT programs - - reformat_User_Code(): - a function to identify python code blocks that aren't properly formated, and auto format the code - for other reddit users. Might live it its own module. - Currently being worked on. - - Leverage reformat user code with automatic Q&A: Use classified code regions to match SO code regions, classified text regions to match SO text regions. Hopefully this improves the search engine and cuts the risk of added noise by a text to code block increasing precieved distance between the user query and the SO database post. Next If a majority of highly matching SO posts have sample code in the question, but the reddit query does not, strongly suggest adding the example code that caused the issue to the next itteration of the query. - question_topic_Modeling(): This is going to take a few parts. - - Identify all related learning subreddits: + - Identify all related learning subreddits using a topic model - Model the topics of stack overflow questions. diff --git a/rpiManager.py b/rpiManager.py index d663c5c..627795b 100644 --- a/rpiManager.py +++ b/rpiManager.py @@ -12,14 +12,15 @@ logging ahead of the main settings, causing logging to just be printed ''' +import sys +from utils import startupLoggingCharacteristics +import logging import subprocess import traceback import socket -import sys import os import time -import logging import threading import datetime @@ -245,12 +246,12 @@ def allBotActions(): if __name__ == "__main__": # Logging Stuff - dirName = "logs" - if not os.path.exists(dirName): - os.makedirs(dirName) - logFileName = 'LOG_'+ datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + '.log' - filePath = os.path.join(dirName, logFileName) - logging.basicConfig(filename=filePath, level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(filename)s:%(funcName)s():%(lineno)s - %(message)s') + # dirName = "logs" + # if not os.path.exists(dirName): + # os.makedirs(dirName) + # logFileName = 'LOG_'+ datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + '.log' + # filePath = os.path.join(dirName, logFileName) + # logging.basicConfig(filename=filePath, level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(filename)s:%(funcName)s():%(lineno)s - %(message)s') # check GPIO Flags # F0: Continue as is diff --git a/utils/botMetrics.py b/utils/botMetrics.py index 061c922..cce1b35 100644 --- a/utils/botMetrics.py +++ b/utils/botMetrics.py @@ -115,7 +115,7 @@ def performanceVisualization(reddit): user = archiveAndUpdateReddit.get_redditor_by_name(reddit, 'pythonHelperBot') totalCommentKarma = user.comment_karma date, karma = archiveAndUpdateReddit.makeCommentKarmaReport(user, reddit) - karmaPlot(date, karma, totalCommentKarma) + #karmaPlot(date, karma, totalCommentKarma) return diff --git a/utils/botSummons.py b/utils/botSummons.py index 42ed8ce..6453324 100644 --- a/utils/botSummons.py +++ b/utils/botSummons.py @@ -100,12 +100,15 @@ def handleInbox(reddit, codeVTextClassifier, phbArcPaths, setOfPosts={}, unreadC commentReplies = 0 directMessages = 0 userNameMentions = 0 + ids_to_comments = [] for msg in inboxMessages: if msg.was_comment: if msg.subject == "username mention": userNameMentions += 1 + ids_to_comments.append(msg.id) else: commentReplies += 1 + ids_to_comments.append(msg.id) else: # Check to see if it was a karma summons directMessages += 1 diff --git a/utils/buildComment.py b/utils/buildComment.py index daa1345..9588212 100644 --- a/utils/buildComment.py +++ b/utils/buildComment.py @@ -16,25 +16,25 @@ def botIntro(): def standardIntro(): msg = '''It looks to me like your post might be better suited for r/learnpython, -a sub geared towards questions and learning more about python. +a sub geared towards questions and learning more about python **regardless of how advanced your question might be**. That said, I am a bot and it is hard to tell.''' return msg def alreadySuggestedComment(): msg = '''I see someone has already suggested going to r/learnpython, -a sub geared towards questions and learning more about python. +a sub geared towards questions and learning more about python **regardless of how advanced your question might be**. I highly recommend posting your question there. ''' return msg def alreadyAnsweredComment(): msg = '''It looks to me like someone might have already answered your question. That said, I am a bot and it is hard to tell. In the future, I suggest asking questions like this in r/learnpython, a sub geared -towards questions and learning more about python. ''' +towards questions and learning more about python **regardless of how advanced your question might be**. ''' return msg def userCrossPosted(): # Figure out what to say to the spray and pray msg = '''It looks like you posted this in multiple subs in a short period of time. In the future, I suggest asking questions like this in learning focused subs like -r/learnpython, a sub geared towards questions and learning more about python. ''' +r/learnpython, a sub geared towards questions and learning more about python **regardless of how advanced your question might be**. ''' return msg def commented_on_before(): From ed9a8262d2895ce05037ddde5819ca75e7044bc4 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Wed, 15 Jul 2020 15:34:42 -0500 Subject: [PATCH 07/10] A error prone Naive Bayes classifier has been added and the state of the bot is acceptable though non ideal. This is the pre alpha version 0.4.00 --- CHANGELOG.md | 105 +++- README.md | 2 +- ROADMAP.md | 46 +- main.py | 45 +- rpiManager.py | 4 +- utils/fix_json_archive_bug.py | 58 +++ utils/formatCode.py | 93 +++- utils/learningSubmissionClassifiers.py | 1 + utils/nb_text_classifier.py | 505 ++++++++++++++++++ utils/nb_text_classifier_2.py | 676 +++++++++++++++++++++++++ 10 files changed, 1491 insertions(+), 44 deletions(-) create mode 100644 utils/fix_json_archive_bug.py create mode 100644 utils/nb_text_classifier.py create mode 100644 utils/nb_text_classifier_2.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 233d188..88712ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,9 +18,9 @@ Dates follow YYYY-MM-DD format -## [A0.4.00] 2020-XX-XX +## [A0.4.00] 2020-07-15 -In Progress +Official. ### Contributors Keith Murray @@ -45,41 +45,64 @@ Logging has been changed to reduce notes from modules outside of the bot (praw a Some tests have also been added, however it does not cover a majority of the bot yet. +A new classifier has been introduced and is being integrated into the bot. It's built off of a naive bayes classifier, but is aimed at classifying based on sentence structure. It's an ugly bodge but its proof of concept seems functional. Further information can be found below and in the header of the file, `nb_text_classifier.py` under utils. + +Off of the new classifier, a larger naive bayesian has been built which is generally more functional. It has a fairly low false negative rate, but far too high of a false positive rate. Both classifiers have been added and commented out in this version. + + + #### Added - - check_for_help_flair in main: a function that looks to see if the new flair is being used on the submission and if it's 'Help'. - - grab_set_of_submissions_flair in main: given a set of submissions, returns a dictionary with their id as the key, and their attached flair as the value - - check_for_help_flair_update: as users can put flair on after they post, the bot needs to be able to compare the old flair with the new flair to see if a 'Help' flair has been added. This check currently runs for two hours after a post has been made. + - `check_for_help_flair` in main: a function that looks to see if the new flair is being used on the submission and if it's 'Help'. + - `grab_set_of_submissions_flair` in main: given a set of submissions, returns a dictionary with their id as the key, and their attached flair as the value + - `check_for_help_flair_update`: as users can put flair on after they post, the bot needs to be able to compare the old flair with the new flair to see if a 'Help' flair has been added. This check currently runs for two hours after a post has been made. - The bot now should auto reply to help flair added within two hours of the original submission, using the previously listed tools to keep track - - In utils, startupLoggingCharacteristics now exists. It adjusts non-python helper bot logs to warning level, and sets up the structure of the logs thereafter, defining the log structure use to be covered in main. - - Added logging info in archiveAndUpdateReddit's comment_duplication_by_ratelimit_check to make it more useful on review. It's still unclear if this patch solves the issue but this logging should make it easier to notice. + - In utils, `startupLoggingCharacteristics` now exists. It adjusts non-python helper bot logs to warning level, and sets up the structure of the logs thereafter, defining the log structure use to be covered in main. + - Added logging info in `archiveAndUpdateReddit`'s `comment_duplication_by_ratelimit_check` to make it more useful on review. It's still unclear if this patch solves the issue but this logging should make it easier to notice. + - Added `nb_text_classifier.py`: a classifier which uses naive bayes over word pairs to calculate whether or not the title of a post is probably learning. It will get torn apart and reworked going foward, but the proof of concept might be worth integrating into the bot right now. Future work will look at a much more expansive naive bayes classifier over all the post features, so when it achieves a specific confidence it'll auto comment, independent of the age of the post. + - In `formatCode`: `astAndCodeopClassifications` uses modules `ast` and `codeop` to classify lines as code or not: requires their syntax to be valid. This may completely replace the classifier passed into `formatCode` as a whole. + - in `formatCode`: `remove_code_blocks` is added which remaps the code into a string '' + #### Changed - Updated Year in License - Added Flair auto-reply explaination in README, de-emphasized bot summons to reflect the lack of active development, added a pre-alpha goal of a reddit post classifier, and adjusted the ethics section to acknowledge the newly allowed rule of commenting on a user multiple times under specific conditions. - Expanded on FAQ - - Logging should now be more strongly defined by this bot, not sessions and connectionpool, hopefully that'll clean up bug hunts a bit. - - Adjusted the format your code comment to make it more noticable - - In botHelperFunctions logPostFeatures: it now records flair - - In learningSubmissionClassifiers, basicUserClassify now is allowed to comment on a post by a user who had a prior post the bot commented on IF that user uses the 'Help' flair. - - In learningSubmissionClassifiers, basicUserClassify: Bot is now allowed to comment even if user has directed others to learnpython before in the past. - - In botHelperFunctions, added submission_flair_text to the logged post features - - Reworded a lot of the bot's comments in buildComments to emphasize that answers aren't instantanious and added emphasis to the format your code link, since that's the most often ignored aspect of users to follow the bots direction. + - Logging should now be more strongly defined by this bot, not `sessions` and `connectionpool`, hopefully that'll clean up bug hunts a bit. + - Adjusted the 'format your code comment' to make it more noticable + - In `botHelperFunctions` `logPostFeatures`: it now records flair + - In `learningSubmissionClassifiers`, `basicUserClassify` now is allowed to comment on a post by a user who had a prior post the bot commented on IF that user uses the 'Help' flair. + - In `learningSubmissionClassifiers`, `basicUserClassify`: Bot is now allowed to comment even if user has directed others to learnpython before in the past. + - In `botHelperFunctions`, added `submission_flair_text` to the logged post features + - Reworded a lot of the bot's comments in `buildComments` to emphasize that answers aren't instantanious and added emphasis to the format your code link, since that's the most often ignored aspect of users to follow the bots direction. - Added a comment that r/learnpython is the place for questions regardless of how advanced the question is. + - `classifyPostLines` in `formatCode.py` now also takes in `astAndCodeopClassifications`'s classification for each line and returns it. + - `alreadyCorrectlyFormatted` in `formatCode.py` takes in the astClassificaiton (it's really the pair of ast and codeop plus a hand written (artisanal?) dedent classification), and it classifies a line as code if the rwc, classifier, or astClassifier calls it code. (Given that the classifier--a simple naive bayes classifier is a bit overzelous--it might be adjusted or removed) #### Deprecated - Commented out the call to karmaPlot in botMetrics. It'll need to be more fully removed later on, but the bot no longer needs it. + - Not yet deprecated, but moving towards it: the classifier built by `buildTextCodeClassifier` in `formatCode.py` might be removed because it might be replaced by the `astAndCodeopClassifications` classifier. Use this as a chance to build an evaluation metric for this feature as a whole. #### Removed #### Fixed - - Under archiveAndUpdateReddit, when evaluating a submission, the bot checks the user of a post and grabs and rewraps it into the wrapper user class. On rare occasions, if a post would be deleted at just the right time, the bot would populate info for the post, the post would be deleted, then the bot would try to populate info for the user which is no longer tied to the now deleted/removed post. A try/except block has been added and now the submissionList waits until both the post and the user info has been successfully built out before it adds it to the list. + - Under `archiveAndUpdateReddit`, when evaluating a submission, the bot checks the user of a post and grabs and rewraps it into the wrapper user class. On rare occasions, if a post would be deleted at just the right time, the bot would populate info for the post, the post would be deleted, then the bot would try to populate info for the user which is no longer tied to the now deleted/removed post. A try/except block has been added and now the submissionList waits until both the post and the user info has been successfully built out before it adds it to the list. #### Security +#### Tests + - Adding comment tree tests: basically purpose different inputs into buildHelpfulComment under main, and ensure the string needed is present + - Added test to make sure test_check_for_help_tag() functions + + ### Main - - The whole logging structure has been updated so that it's now defined in utils/startupLogginCharacteristics. This helps clean up the main and makes it easier to maintain a logging format through the program, preserve and transfer the format to other projects, and suppress uninformative logging messages by imported modules. - - check_for_help_flair, grab_set_of_submissions_flair, and check_for_help_flair_update have been added as have calls to these functions. This collection of functions helps track flair on posts as they've been added as it is not usually added by users right away. The most common reason the bot misses help flair is if the flair was added more than two hours after the initial submission time--reddit rewrites the post age to reflect the age it should be after subtracting the amount of time the submission was removed for. The bot assumes this doesn't happen and may need another change soon to address this issue as well. + - The whole logging structure has been updated so that it's now defined in `utils`/`startupLogginCharacteristics`. This helps clean up the main and makes it easier to maintain a logging format through the program, preserve and transfer the format to other projects, and suppress uninformative logging messages by imported modules. + - `check_for_help_flair`, `grab_set_of_submissions_flair`, and `check_for_help_flair_update` have been added as have calls to these functions. This collection of functions helps track flair on posts as they've been added as it is not usually added by users right away. The most common reason the bot misses help flair is if the flair was added more than two hours after the initial submission time--reddit rewrites the post age to reflect the age it should be after subtracting the amount of time the submission was removed for. The bot assumes this doesn't happen and may need another change soon to address this issue as well. - It now checks flair for the full 24 hours it watches a submission. -### rpiManager.py + - `nb_text_classifier` and `nb_text_classifier_2` are imported in an experimental capacity + - `lookForKeyPhrasePosts` now takes the naive bayes classifier, `nb_submission_classifier` as an input variable. From here it classifies the post as boolean: `learning_title`, and adds that submission to the list `submissionsToCommentOn_KP + - `title_classifier` is the function which evalutes a submission on the naive bayes classifier side of things. Currently there's a lot of printing so I can see how it's working. + - Startup loads the naive bayes classifiers, only the full submission classifier is actually returned (`nb_text_classifier_2`). Similarly run bot takes it as input, and `if __name__ == "__main__":` handles the return and function call. +### rpiManager.py + - `botStuff` loads the naive bayes classifiers, only the full submission classifier is actually returned (`nb_text_classifier_2`). Similarly run bot takes it as input, and `if __name__ == "__main__":` handles the return and function call. ### Util Libraries @@ -94,12 +117,54 @@ Some tests have also been added, however it does not cover a majority of the bot - commented_on_before has been added to acknowledge that users have interacted with it before. (This should only activate during help flair--we'll see) - followSubRules has been adjusted to remind users that people take time to answer questions, and they may not get a reply right away - formatCodeAndOS has been adjusted to emphasize the link to how to format your code +#### fix_json_archive_bug.py + - Created to handle trailing comments in the archive which were mistakenly given a new file due to a bug in the file spliter of `saveClassJson`. This is not yet functional and will not be used by the bot, but by an archive manager. #### formatBagOfSentences.py #### formatCode.py + - Now imports `ast` and `codeop` + - `astAndCodeopClassifications` now exists. It takes in a line of text. This just checks whether or not a line has valid syntax for python code. + - `astAndCodeopClassifications` is now called by the rewrapClassifications function. + - `alreadyCorrectlyFormatted` now declares code is present if any line is called code by any of the three code classifiers. + - function `remove_code_blocks` has been added. If a line is called 'code' by either the rwc or astc, then it is replaced with a flag. Blocks of flags are stripped into a single replacement flag, `CODE` for text processing. #### learningSubmissionClassifiers.py - As a result of this version's changes, basic user classify now only prevents comments on users if they've been commented on by the bot before AND are not using the 'Help' Flair. Given this trend, it's reasonable to assume the bot will soon move to comment on users regardless of it's past history with them, as byinlarge the bot is usually correct within a reasonable degree when it's other classifiers activate. #### locateDB.py #### lsalib2.py +#### nb_text_classifier.py +This module was added in full in this version. + +It will be removed and replaced by `nb_text_classifier_2`. It was a proof of concept to see how well the naive bayesian classifier worked on titles of submissions. The prior experiment which failed, but built towards this was an LDA topic model, which never picked up on + +The module explains in depth what it does, so we'll briefly address its shortfalls here. + +The model is pretty useful, getting a solid true positive rate, but always hitting false positive on anything that has a structure similar to "Data manipulation with Numpy": titles which just state the subject. Half the time posts of that style are showcases of projects or explainations about that subject, and the other half of the time they're questions centered on the topic. + +The model in `nb_text_classifier_2` has this same shortfall. + +Specific to `nb_text_classifier`, the code is written in a way that's difficult to extend to a full submission classifier, as well as difficult to tweak to test with different thresholds. It remains as a solid first shot which is why it is being committed, but it almost certainly wont be preserved past this. + + +#### nb_text_classifier_2.py +The key change to this classifier is the ability to add various submission features to the classifier. Though it's large shortfall is the lack of using the LogSumExp trick to handle that addition. + +Using Aziraphale (personal database manager) and `mod_judged_posts` to gather posts from the archive, it splits the archive into a train and test set, and 'successful' and 'learning' classes. + +It is passed to the reddit submission classifier class (`reddit_submission_classifier`) which splits the 'judged' posts into two categories and an undefined collection, 'Learning', 'Successful', and 'unclassified'. Submission features are broken down to + +Features it currently evaluates between the classes are: Title word pairs, selftext word pairs, and link type (selftext, outlink, image, video, crosspost). + +The `reddit_submission_classifier` splits the submissions into the three groups, and passes the labeled classes ('successful' and 'learning') into `build_classifiers` one by one, which populates `reddit_submission_class_fields_naivebayes`. Titles and selftext are preprocessed by `text_preprocessor` and link types are preprocessed by `reddit_link_simplifications` then placed into `simple_feature`. + +After the `reddit_submission_class_fields_naivebayes` object is built for each class, you can pass a submission to each one using the `get_p_of_submission_in_class` attribute to get the probability that some given submission alpha is generated by the distrobution in the model for that class (be it the learning or successful class). + +The parent class, `get_p_of_submission_in_class` will classify a submission by getting the likelhood it belongs to both models, and classifying the submission according to which class has a greater negative log likelyhood. It then creates a hand wavey confidence score based on the difference between the two most likely classes normalized by the largest negative log likelyhood. This isn't valid science, but this was a proof of concept and I needed to get a number value which made some tactile sense that I could use as a threshold for letting the bot comment. + +Under the current version, confidence is defined as `(np.log(np.abs(score_diff)))/(-s_score)` where `score_diff` is the difference between the two most likely classes (and since there's only two classes--successful or learning--it's the difference between those two), and `s_score` is the negative log likelyhood of the most likely class. From there the bot was 'allowed' (in a testing/quiet mode capacity) to comment if the confidence was greater than 0.004 and the s_score was greater than -1000. Again, these are hand wavy values only used to get a sense of whether or not the classifier needed more work. Which, in short it does. + +`get_p_of_submission_in_class`, an attribute of `reddit_submission_class_fields_naivebayes` needs to use the LogSumExp trick instead of a simple summation since the presence of selftext, and the content of that selftext is an attribute of the link type feature: so it can't simply be summed with the same weight as title text, or have the same impact as linking to a blog post. There was a hope that this statistical sin would still be functionally allowable, but that isn't the case. + +In all, the code will remain, the logsumexp trick shouldn't be rough to add, and it was a useful improvement off of the `nb_text_classifier` file. + #### questionIdentifier.py #### rpiGPIOFunctions.py #### scriptedReply.py @@ -110,9 +175,7 @@ Some tests have also been added, however it does not cover a majority of the bot #### updateLocalSubHistory.py #### user_agents.py -### Tests - - Adding comment tree tests: basically purpose different inputs into buildHelpfulComment under main, and ensure the string needed is present - - Added test to make sure test_check_for_help_tag() functions +### Tests ## [A0.3.02] 2019-09-16 diff --git a/README.md b/README.md index 9930841..92fe814 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ The bot is currently in a pre alpha stage. This means that the founding goals of - [X] Simply comment on redditors posts who look like they should post in r/learnpython - [X] Run on the raspberry pi - [X] Archive reddit posts for future classification evaluation - - Build a reddit post classifier using the archive and LDA + - [X] Build a reddit post classifier using the archive and Naive Bayes - Build a local Stack Overflow Search Engine - Use Stack Overflow to gauge the simplicity of a redditors question - Use Stack Overflow to implement a naive Question and Answer system diff --git a/ROADMAP.md b/ROADMAP.md index 21d4a9e..6f9e3d1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -23,9 +23,9 @@ Dates follow YYYY-MM-DD format -## [A0.4.00] 2020-XX-XX +## [A0.4.00] 2020-07-15 -In Progress +Official. ### Contributors Keith Murray @@ -38,12 +38,22 @@ Unless otherwise noted, all changes by @kmurrayis ### Short Term Roadmap -Address Flair and begin adding tests. When min_spanning_tree program is complete, as well as when 'fix reddit archive' is done, begin rolling out a new classifier using LDA +[X] Address Flair and begin adding tests. + +When min_spanning_tree program is complete, as well as when 'fix reddit archive' is done, begin rolling out a new classifier using LDA + +Strike the above--LDA was a nice effort, but it didn't classify on 'learning'v'showing off' classes as well as I had liked, even when coaxed to do so with a fairly heavy hand. Instead it grouped things like 'web dev' and 'questions about web dev' together in one class, and 'machine learning' and 'questions about machine learning' in another class, and so on. This is good to know and probably should have been expected but the deviation was worthwhile. + +[X] Instead a new naive bayes classifier has been added and is being integrated into the bot. + +Once the new classifiers are in place, the bot is going to be reworked again, much more completely this time. +Rather than passing individual classifiers around, there will be a classifier group/class that'll be passed, and it'll handle all individual classifiers. Reddit Submission classes will be incorporated into a larger class which holds the reddit features, as well as that submissions classifications. This may add some ram strain, but the group will make the code easier to follow, debug, and add to. It can be optimized later. + + #### Add - - With light archive, build a 'save state' and 'load state' function so records on new posts aren't lost during reboots. + - With light archive, build a 'save state' and 'load state' function so records on new posts aren't lost during reboots. This should help resolve 'posts it saw which didn't have help flair, the bot powered down, the flair was applied, and the bot powered up' order of opperation issue. - botHelperFunctions/botMetrics: Add ram usage check and log it in every 'awake' cycle of program, to try and tease out any possible MemeoryError's kicking up after long periods of time. Might as well save other diagnostics info, maybe call this from the hearbeat thread so it becomes a better representation of runtime - Also a pull origin master from github state would be incredibly useful - [X] Verify reddit post logs by grabbing most recent bot comments. This should reduce risk of two computers commenting on the same post which could happen if databases are de-synced and one computer is not in quiet mode because I typed in the wrong command. Opperator Error Risk Reduction. Only the most recent x+buffer hours of interaction are needed. This does not protect from multiple posts by the same user but should prevent multiple comments on the same post even if the bot runs on different computers. - Full Test Suite: PRIORITY (Ok it'll have to wait until posts have been archived, but it'll still be nice) - botMetrics.measureUserReaction(): @@ -61,13 +71,14 @@ Address Flair and begin adding tests. When min_spanning_tree program is complete - Move NLP functions from NLTK (used in many files) to a buffer module, allowing for simpler, universal changes to be made. For example, if there is a better POS tagger than nltk.pos_tag(sent) then we can easily switch to that. Right now NLTK is used over a fairly large filespace making adjustments of that sort difficult. - Review all my logging notes. See what should be dropped, changed, etc. - [X] Make sure the bot defaults to commenting about formatting even if there's no code present--It just thinks there's code a touch too often - - If it classifies a line as code, try to validate the expression in a basic form. This might improve the classifier and reduce isThereCode False Positives. + - [X]If it classifies a line as code, try to validate the expression in a basic form. This might improve the classifier and reduce isThereCode False Positives. Used ast and codeop for this. - When errors occur, or a shutdown button is pressed, attempt to save current reddit info into a temp file. on startup, load that info in then connect with praw to expand the knowledge base. - The archive and update reddit module has become unwieldy. It should be broken into two modules, one which handles the recasting of the praw classes, and one which deals with them as needed.--Ehh.. Maybe not. It does need to be together to a degree - Migrate away from usage of my personal libraries so it's easier for others to get the bot up and running. #### Deprecate + - [X] karma scatter plot. It's no longer useful or very interesting. #### Remove @@ -96,8 +107,6 @@ Address Flair and begin adding tests. When min_spanning_tree program is complete This includes "How do I install python" and "Is learning python worth it?" (Aka "Why learn python?")--A quality version of this is a full research task. Build off of internal sentence ordering models + soft skills. Pull text from comments on these posts to build up scripted reply. Look into adapting this (or maybe starting with this) to classifying the posts so their types can be diplayed as tags. - When LDA classifier comes into play, this might be a lot easier. Auto-segment posts into topics, ID topic of question, compare question to similar past question, map past answers to new question. - - local Flask website/dashboard to monitor the status and logs of the bot in realtime. Low priority. - - Log processing: a set of functions and visualizations to process the log files for various useful tidbits. Something nicer than grep @@ -180,6 +189,7 @@ other areas - Previous code is stored in a tree like structure - Leverage sentence ordering ideology to say given the current line and the previous state of the code tree, which level of node in the tree should I be This should be an area of linguists where there's plenty of work already completed, look for it. I think Nevil-manning sequitor addresses it briefly, look at that+cited by for other work in the area. + - This might be done with the abstract syntax tree module ### learningSubmissionClassifiers.py @@ -198,6 +208,26 @@ other areas ### lsalib2.py Migrate features back into lsalib +### nb_text_classifier.py +Needs to be deprecated and removed + +It will need to be cleaned up and swapped out from this frankenstein code and moved to use a more legitimate library. It should also use the same format so the presence of selftext, a link to i.reddit, or a link to a third party site can be added to the calculation, as well as have all of those features added without having to completly rework the core code. + +For selftext posts, consider another weird classification: + break the text into blocks then sentences + As was considered with selftext prior, remap all code to CODE, and merge all neighboring instances of code into one block. + classify each sentence: maybe use LDA to generate m topics, and make a m space. + Final classification for the selftext will be the probility that a question post + built sentences which progressed in that way. This way a rhetorical question is + less likely to mess it up. + + +### nb_text_classifier_2.py +`get_p_of_submission_in_class` needs to use logSumExp trick. + + + + ### questionIdentifier.py It'd be nice to use stack overflow's user submissions and r/learnpython's submissions compared to 'successful' r/python submissions to build a 'programmers diff --git a/main.py b/main.py index e5feb97..2c92890 100644 --- a/main.py +++ b/main.py @@ -19,6 +19,8 @@ from utils import formatCode from utils import learningSubmissionClassifiers from utils import locateDB +from utils import nb_text_classifier +from utils import nb_text_classifier_2 from utils import questionIdentifier from utils import searchStackOverflowWeb from utils import summarizeText @@ -203,7 +205,7 @@ def check_for_key_phrase(submission, phrase_set): request_Made = learningSubmissionClassifiers.request_Key_Word_Classifier(submission, phrase_set) return request_Made -def lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set): +def lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set, nb_submission_classifier): oldPosts = setOfPosts.copy() # https://stackoverflow.com/questions/5861498/ setOfPosts = archiveAndUpdateReddit.getNewPosts(reddit, submissionList=setOfPosts) @@ -212,8 +214,9 @@ def lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set): if key not in oldPosts: submission, user = setOfPosts[key] request_Made = check_for_key_phrase(submission, phrase_set) + learning_title = title_classifier(submission, nb_submission_classifier) help_tag = check_for_help_flair(submission) - if request_Made or help_tag: + if request_Made or help_tag or learning_title: submissionsToCommentOn_KP.append(key) return setOfPosts, submissionsToCommentOn_KP @@ -225,6 +228,30 @@ def basicQuestion_classifyPost(submission, classifier): question_Sents = learningSubmissionClassifiers.basicQuestionClassify(submission, classifier) return question_Sents +def title_classifier(submission, nb_submission_classifier): + ''' + In the current version, learning posts are class 0 + successful posts are class 1 + ''' + c = nb_submission_classifier.classify_submission(submission) + className = 'successful' if c else 'learning' + logging.debug("Title classified as '"+className.capitalize()+\ + "' With Confidence: "+str(nb_submission_classifier._confidence)+\ + " and with NegLogLikelyhood: "+str(nb_submission_classifier._score)) + print("* ",submission.title) + print("* ",submission.id) + print("* Title classified as '"+className.capitalize()+\ + "' With Confidence: "+str(nb_submission_classifier._confidence)+\ + " and with NegLogLikelyhood: "+str(nb_submission_classifier._score)) + if c == 0 and nb_submission_classifier._confidence > 0.004 and nb_submission_classifier._score > -1000: + logging.debug("Title was strongly classified as learning") + print("* Title was strongly classified as learning") + print("*"*30) + return True + print("*"*30) + + return False + def handleSetOfSubmissions(reddit, setOfPosts, postHistory, classifier): submissionsToCommentOn_BC = [] @@ -303,6 +330,10 @@ def startupBot(): classifier = questionIdentifier.buildClassifier02NLTKChat() # Code vs Text classifier codeVTextClassifier = formatCode.buildTextCodeClassifier(sourceDataPath=paths["codeText"]) + # Naive Bayes Title Classifier + nb_title_classifier = nb_text_classifier.Naive_Bayes_Title_word_pair_classifier() + nb_submission_classifier = nb_text_classifier_2.build_reddit_submission_classifier() + # Reddit API keySet = getPythonHelperBotKeys.GETREDDIT() @@ -319,11 +350,11 @@ def startupBot(): logging.debug( "Loaded. Running...") - return reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, phbArcPaths + return reddit, classifier, codeVTextClassifier, nb_submission_classifier, tdm, userNames, postHistory, phbArcPaths -def runBot(reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, phbArcPaths={}, quietMode=False): +def runBot(reddit, classifier, codeVTextClassifier, nb_submission_classifier, tdm, userNames, postHistory, phbArcPaths={}, quietMode=False): phrase_set = botHelperFunctions.load_autoreply_key_phrases(fl_path='misc/autoreplyKeyPhrases.txt') @@ -353,7 +384,7 @@ def runBot(reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, commentOnThese += submissionsToCommentOn_HF # Get new posts, respond to keywords - setOfPosts, submissionsToCommentOn = lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set) + setOfPosts, submissionsToCommentOn = lookForKeyPhrasePosts(reddit, setOfPosts, phrase_set, nb_submission_classifier) commentOnThese += submissionsToCommentOn lastThreeMin = datetime.datetime.now() @@ -430,9 +461,9 @@ def interface(): if quietMode: logging.debug("Running in Quiet Mode") - reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, phbArcPaths = startupBot() + reddit, classifier, codeVTextClassifier, nb_submission_classifier, tdm, userNames, postHistory, phbArcPaths = startupBot() try: - runBot(reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, phbArcPaths=phbArcPaths, quietMode=quietMode) + runBot(reddit, classifier, codeVTextClassifier, nb_submission_classifier, tdm, userNames, postHistory, phbArcPaths=phbArcPaths, quietMode=quietMode) except KeyboardInterrupt: print("Concluding Program") logging.debug("Keyboard Interrupt: Ending Program") diff --git a/rpiManager.py b/rpiManager.py index 627795b..5fb074c 100644 --- a/rpiManager.py +++ b/rpiManager.py @@ -171,9 +171,9 @@ def pull_from_github(): def botStuff(): time.sleep(30) - reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, phbArcPaths = main.startupBot() + reddit, classifier, codeVTextClassifier, nb_title_classifier, tdm, userNames, postHistory, phbArcPaths = main.startupBot() try: - main.runBot(reddit, classifier, codeVTextClassifier, tdm, userNames, postHistory, phbArcPaths=phbArcPaths) + main.runBot(reddit, classifier, codeVTextClassifier, nb_title_classifier, tdm, userNames, postHistory, phbArcPaths=phbArcPaths) except KeyboardInterrupt: print("Concluding Program") logging.debug("Keyboard Interrupt: Ending Program") diff --git a/utils/fix_json_archive_bug.py b/utils/fix_json_archive_bug.py new file mode 100644 index 0000000..ae393c1 --- /dev/null +++ b/utils/fix_json_archive_bug.py @@ -0,0 +1,58 @@ + + + + +''' +TinyFile Bug: + every new dictionary has a '[' ahead of it instead of a comma + +Large File Bug: + Either + Multiple ']' at the end of the file + Or + Missing trailing ']' +''' + +import json + + +import os + +def fix_improper_restart_bug(text): + ''' + Occasionally the json will append the file + as if it's a new json file, }[\\n rather than + },\\n + + So this will simple replace it + ''' + text = text.split('}[\n') + text = '},\n'.join(text) + return text + +def fix_trailing_brackets_bug(text): + ''' + Some of the files get multiple ']' tacked onto + the end of them, when they just need one closing bracket + + So we'll strip them out and try to tack one on + ''' + while text.rstrip()[-2:] == '\n]': + text = text[:-2] + + if text.rstrip()[-1] != ']': + text += '\n]' + return text + + +def fix_reddit_json_bug(filepath): + with open(filepath, 'r') as ifl: + text = ifl.read() + text = fix_improper_restart_bug(text) + text = fix_trailing_brackets_bug(text) + text_struct = json.loads(text) + + + return + + \ No newline at end of file diff --git a/utils/formatCode.py b/utils/formatCode.py index 6419658..a9d8a44 100644 --- a/utils/formatCode.py +++ b/utils/formatCode.py @@ -11,6 +11,8 @@ from nltk.tokenize.treebank import TreebankWordTokenizer, TreebankWordDetokenizer from nltk.corpus import brown from nltk.metrics import ConfusionMatrix +import ast +import codeop from utils import archiveAndUpdateReddit @@ -97,6 +99,7 @@ def buildTextCodeClassifier(sourceDataPath): Takes in a training file filled with coding samples and uses the NLTK provided Brown News sentence corpus and builds + MIGHT SOON BE DEPRECATED ''' randomSeed = random.randint(0,1000) @@ -149,6 +152,44 @@ def buildTextCodeClassifier(sourceDataPath): return classifier#, train_set, test_set +def astAndCodeopClassifications(line): + ''' + Checks to see if a line of text is valid python syntax + + The two modules it uses are `ast` and `codeop`, both a part of the + standard library. + This program checks the line first by using `ast.parse`. + If ast throws a syntax error, it tries codeop. + If that fails as well, it's deemed not code, although it should be + noted it could be (invalid) code which is only a couple of character + translations away from being valid. + ''' + dedent_triggers = ['elif', 'else', 'except'] + dedent_triggers += [x+':' for x in dedent_triggers] + line=line.strip() # Needed to remove whitespace + if line.strip() in ['​', '']: + return 'emptyline' + try: + ast.parse(line.strip()) + c = 'code' + except SyntaxError: + # Possible it's still code, just the opening of a statement + try: + comp = codeop.compile_command(line) + c = 'code' # regardless of comp being none or code object + except SyntaxError: + tokens = line.split() + if tokens[0] in dedent_triggers and line[-1] == ':': + # Need to manually handle dedent case, kinda hand wavey + c = 'code' + else: + c = 'text' + except ValueError: + # Possible output, not sure how it's triggered though + # https://docs.python.org/3/library/codeop.html#codeop.compile_command + c = 'text' + + return c def rewrapClassifications(line): @@ -258,11 +299,13 @@ def classifyPostLines(textBlock, classifier): lines = textBlock.split('\n') classifications = [] rwclassifications = [] + astClassifieds = [] for line in lines: c = classifier.classify(code_text_features(line)) classifications.append(c) rwc = rewrapClassifications(line) rwclassifications.append(rwc) + astClassifieds.append(astAndCodeopClassifications(line)) #print(c, rwc, line) ''' print('\t>',c, line) @@ -270,10 +313,10 @@ def classifyPostLines(textBlock, classifier): for label in dist.samples(): print("\t\t>%s: %f" % (label, dist.prob(label))) ''' - return classifications, rwclassifications + return classifications, rwclassifications, astClassifieds -def alreadyCorrectlyFormatted(c, rwc): +def alreadyCorrectlyFormatted(c, rwc, astc): ''' Function itterates through the by rewrapped classifications and compares them to the naive bayes classifications. If there @@ -287,7 +330,7 @@ def alreadyCorrectlyFormatted(c, rwc): codePresent = False correctlyFormatted = True for i in range(len(rwc)): - if rwc[i] == 'code' or c[i] == 'code': + if rwc[i] == 'code' or c[i] == 'code' or astc[i] == 'code': codePresent = True if rwc[i] == 'NA' and c[i] == 'code': correctlyFormatted = False @@ -309,8 +352,8 @@ def reformat(text, classifier): """ # Classify the lines in the text - c, rwc = classifyPostLines(text, classifier) - codePresent, correctlyFormatted = alreadyCorrectlyFormatted(c, rwc) + c, rwc, astc = classifyPostLines(text, classifier) + codePresent, correctlyFormatted = alreadyCorrectlyFormatted(c, rwc, astc) sourceText = text.split('\n') #print(len(sourceText), len(c), len(rwc)) assert (len(sourceText) == len(c)) and (len(c) == len(rwc)) @@ -365,6 +408,46 @@ def reformat(text, classifier): #logging.info("Code Present: " + str(codePresent) + " | Correctly Formatted" + str(correctlyFormatted)) return msg, changesMade, codePresent, correctlyFormatted +def remove_code_blocks(text, classifier): + ''' + Strip out all lines of code for the bot to rework + as [code block] for the selftext naive bayes classifier + + Returns the text of the post with the code blocks replaced by + the key string 'CODE' + ''' + code_blocked_text = '' + + # Classify the lines in the text + c, rwc, astc = classifyPostLines(text, classifier) + text = text.split('\n') + + line_types = [] + for i in range(len(c)): + if rwc[i] in ['codeblock'] or astc[i] == 'code': + line_types.append('c') + elif rwc[i] == 'emptyline': + if i > 0: + line_types.append(line_types[-1]) + else: + line_types.append('t') + else: + line_types.append('t') + + cb = False + for i in range(len(text)): + if line_types[i] == 'c': + if cb == False: + code_blocked_text += 'CODE\n' + cb = True + else: + code_blocked_text += text[i] + '\n' + + + + return code_blocked_text + + def loadSummoningHistory(sourcefl): submissions = {} reformatted = [] diff --git a/utils/learningSubmissionClassifiers.py b/utils/learningSubmissionClassifiers.py index 1725169..6836e2d 100644 --- a/utils/learningSubmissionClassifiers.py +++ b/utils/learningSubmissionClassifiers.py @@ -13,6 +13,7 @@ from utils import buildComment from utils import formatCode from utils import locateDB +from utils import nb_text_classifier from utils import questionIdentifier from utils import searchStackOverflowWeb from utils import summarizeText diff --git a/utils/nb_text_classifier.py b/utils/nb_text_classifier.py new file mode 100644 index 0000000..2f51c16 --- /dev/null +++ b/utils/nb_text_classifier.py @@ -0,0 +1,505 @@ + + + + + +import numpy as np +import nltk +import datetime +import logging +import json + +''' +An early outline of a naive bayes classifier for learning posts +which relies on word pairs to make the classification. + +There are better implementations of this classifier, and in the +future the should be reworked to use them instead. None the less +this was the code that satasified the proof of concept, and so as +to not let perfect be the enemy of good, we're rolling with this. +Below is a detailed explaination of what I'm doing so I can return +to this much later and remember what on earth I did to get these +results. + +The code is divided into three sections plus one + +Build Classifier +Save Classifier +Load Classifier + +And Classify as the final 'plus one' section + + + +# About this classifier +--------------------- +This classifier was built after an attemped to use LDA as a means +to classify learning/question posts as if learning was a topic. +That didn't function. + +The hypothesis is that topic model failed largely because it +downweighted the key +elements of a sentence which identify it as a question. +These features are common parts of speech and word order. +From there, it's guessed that word order can be measured by +word pairs, and to reduce the space of all sentences which can be used +only the top k words will be preserved. This ensures the classifier uses +the key features which get downweighted in LDA and LSA models. + +The classifier is fairly custom and will need some work to be more generalizable +but that's more or less ok as this is a proof of concept. It's built to expect +two classes, successful, and learning (This is because the bot exists to id learning +posts, all else are unimportant to it). +It should be added to other naive bayes classifiers to incoperate things like +p (learning | link[outsite]/link[i.reddit]/selfpost), and so on and so forth + + + + +## Gathering the data +Since around march 2019, the bot has been archiving most of the reddit submissions +the r/python. + +Over all of the posts recorded, the top 1k words used in the titles were recorded. + +To build the classifier, there needs to be a set of posts identified as learning, +and a set of posts identified as 'successful': a hand wavey way to say, 'things +the bot shouldn't interact with'. +Because there's a bug in the archive, the model for the confusion matrix +outlined in the ROADMAP is a bit difficult to extract. +(The comments aren't currently easily associated to the submissions) +But the flair: 'removed: Learning' is still a valid measure. + +Going off of the removed posts, it's assumed that all posts made 6 hours before the +post which was later removed by a mod were veiwed by the mods and allowed to stay. +This 6 hour window generates a weak set to draw priors from that defines what is +and is not a learning post. +From that weak prior posts set, all posts which were removed for learning +were placed in a list of learning posts. +All other posts which were not removed and scored above some threshold of karma +(6 in this version) were deemed successful posts. +This left a set of 'unclassified' posts. +In the future when the comments are attached to their parent post, this will be +redone to use comments about posting in r/learnpython as further aid in unsupervized +classification. +(Does crowd sourcing the labeling process mean that the model is supervized? +Or is it unsupervized because the 'crowd' doesn't know they're labeling the data, +and I'm not individually validating the labels? +It feels like it's unsupervized to me, we'll go with that until I'm corrected) + + +## Building the Model + +The model is a naive bayes classifier which uses word pairs to draw the class. +(Future work should look at extending this past word pairs using an LZ dictionary +design or sequitor like structure) +For two reasons, this model restricts the words to the top 1k words used in reddit +submission titles over the past year +(This 1k word list is taken before the learning class list is build, and includes +posts outside of either the learning post list or the successful post list) +The first reason is to reduce the space of the model: + $1k^{2}$ is a lot less than $13k^{2}$ +The second reason is to help the model focus on the structure of the sentence over +the content of the sentence, much of which may be unseen previously (new library name) +while maintaining the possibility of having keywords which help define a post be caught + +If a word isn't in the 1k most used set, it's part of speech tag is substitued into +the string in its place (using tokenization by nltk.word_tokenize or space characters). +The part of speech is determined by the default nltk.pos_tagger, which uses the +penn_treebank pos values. This sets the state space to $1036^{2}$ possible word pairs. + +The priors for `learning` and `successful` posts were taken as the size of their lists +over the total number of posts in the weak prior posts list. +Because both are a subset of the generated weak prior post list, there's a solid +amount of 'undefined' space (around 70% depending on what the karma threshold is +set to for the succesfull posts) + +The count of word pairs is taken over each class, and the total number of pairs is +also recorded so the $P(w_{a},w_{b}|Class_{\\alpha})$ can be calculated on demand, +and more easily updated. + +## Running the model +With the class' priors known, and the $P(w_{a},w_{b}|Class_{\\alpha})$ being ready to +calculate on demand, a string $S$ is classified by: + +Calculating $log(P(Class_{\\alpha}|S))$ for each class by: + + 1. Tokenizing the text, (or spliting by spacecharacters if the tokenization runs + into an error) + 2. Replacing all tokens not in the approved 'common words' list with their part of + speech (using `nltk.pos_tag`) + 3. Begin a rolling sum by taking the negative log likelyhood of the class prior + 4. For Each pair of words `a`, `b`, in the sentence: + Calculate the probability of the word pair in the class by checking: + * If that pair is in the class model: return the $P(w_{a},w_{b}|Class_{\\alpha})$ + * else: return $\\frac{1}{wordCount^{2}*Modifier}$ Where `wordCount` comes from + the total number of common words the model uses plus the part of speech tags, and + the modifier is used to downweight that word pair to less than random + but still non-zero probability + Take the natural log of that value (`np.log()`) to generate the negative log + likelyhood and add it to a rolling sum + 5. Return the final sum of the negative log likelyhood + +When all (in this case both) classes have had their negative log likelyhood calculated, +return the max of the classes. + +A 'confidence' score is also calculated, though it's fairly hand wavey. +The score is the natural log of the difference between the two classes +which can be accessed by `Naive_Bayes_Title_word_pair_classifier.s_confidence`. +The negative log likelyhood score can also be accessed in a similar fashion, +`Naive_Bayes_Title_word_pair_classifier.s_score`. +These values can be used to guage how confident the classifier is that the +string `S` belongs to one class over another and they will be retained until +the next string is classified. + + +''' + +def get_k_most_used_words(posts, k=1000): + ''' + Grab k most used words in the dataset + Anticipates posts to be a custom class + with the title feature present + + ''' + word_c = {} + for post in posts: + words = nltk.word_tokenize(post.title.lower()) + for word in words: + if word in word_c: + word_c[word] += 1 + else: + word_c[word] = 1 + counts = list(word_c.values()) + words = list(word_c.keys()) + large_indicies = np.array(counts).argsort()[-k:][::-1] + max_words = [words[i] for i in large_indicies] + + return max_words + +def rework_sentence(s, allowed_words): + try: + t = nltk.word_tokenize(s.lower()) + except TypeError: + print('Tokenize_error') + t = s.lower().split() + pos = nltk.pos_tag(s) + s_out = [] + for i in range(len(t)): + word = t[i] + if word not in allowed_words: + word = pos[i][1].upper() + s_out.append(word) + s_out = ['START'] + s_out + ['END'] + return s_out + +def mod_judged_posts(reddit_submissions): + + ''' + If a post has been removed by a mod, + all posts for the previous 6 hours are + dumped into out_posts on the assumption + that the mod is active, and has allowed the + previous posts to stay up. + + This creates a weak set of posts to build + priors off of. + ''' + out_posts = [] + posts_buffer = [] + age_limit_hours = 6 + age_limit = datetime.timedelta(hours=age_limit_hours) + learning_post_count = 0 + high_vote_post_count = 0 + + for post in reddit_submissions: + # Add it to the buffer + posts_buffer.append(post) + youngest_post_dt = post.created_utc + pop_posts = 0 + # Pop old posts + for i in range(len(posts_buffer)): + oldest_post_dt = posts_buffer[i].created_utc + if youngest_post_dt-oldest_post_dt < age_limit: + break + posts_buffer = posts_buffer[i:] + # Check if learning post + if not isinstance(post.link_flair_text, type(None)): + if 'removed: Learning' == post.link_flair_text: + out_posts += posts_buffer + posts_buffer = [] + learning_post_count += 1 + + #print(learning_post_count, len(out_posts)) + return out_posts + +def get_word_pair_probability(S, allowed_words, D=None): + ''' + assumes S is a list of strings + + ''' + if isinstance(D, type(None)): + D = {} + word_pair_count = 0 + for s in S: + s = rework_sentence(s, allowed_words) + word_pair_count += len(s)-1 + for i in range(len(s)-1): + pre_word = s[i] # prefix + suf_word = s[i+1] # suffix + if pre_word in D: + pre_d = D[pre_word] + if suf_word in pre_d: + pre_d[suf_word] += 1 + else: + pre_d[suf_word] = 1 + else: + D[pre_word] = {suf_word:1} + + return D, word_pair_count + +def build_model(): + ''' + Must be run on a computer with aziraphale installed + and the training data present. Since this is custom code + it's not expected to work on other machines, but the + results of this build are saved under 'misc/title_classifier.json' + and the load function should work. + ''' + from aziraphale.data_handlers import redditData + from aziraphale.utils import dataSetTools + + handle = redditData.Reddit_Submission_Dataset_Interactions() + reddit_submissions = dataSetTools.DSIterator(handle) + + # Get common words to keep bot from treating a post + # with language it's never seen before as if it were as + # common as can be + max_words = get_k_most_used_words(reddit_submissions, k=1000) + + # Grab only posts which were made while a mod was active + weak_prior_posts = mod_judged_posts(reddit_submissions) + + # From weak prior posts organize it into successful, + # learning, and other--unclassified + learning_posts = [] + successful_posts = [] + unclassified_posts = [] + for post in weak_prior_posts: + if not isinstance(post.link_flair_text, type(None)): + if 'removed: Learning' == post.link_flair_text: + learning_posts.append(post) + elif post.score >= 6: + successful_posts.append(post) + + else: + unclassified_posts.append(post) + + learning_prior = len(learning_posts)/len(weak_prior_posts) + successful_prior = len(successful_posts)/len(weak_prior_posts) + + learning_S = [post.title for post in learning_posts] + successful_S = [post.title for post in successful_posts] + + learning_word_pairs, learning_pair_count = \ + get_word_pair_probability(learning_S, max_words) + successful_word_pairs, successful_pair_count = \ + get_word_pair_probability(successful_S, max_words) + + + return max_words, learning_prior, successful_prior, \ + learning_word_pairs, learning_pair_count, \ + successful_word_pairs, successful_pair_count + + +def build_selftext_model(): + ''' + Must be run on a computer with aziraphale installed + and the training data present. Since this is custom code + it's not expected to work on other machines, but the + results of this build are saved under 'misc/title_classifier.json' + and the load function should work. + ''' + from aziraphale.data_handlers import redditData + from aziraphale.utils import dataSetTools + + handle = redditData.Reddit_Submission_Dataset_Interactions() + reddit_submissions = dataSetTools.DSIterator(handle) + + # Get common words to keep bot from treating a post + # with language it's never seen before as if it were as + # common as can be + max_words = get_k_most_used_words(reddit_submissions, k=1000) + + # Grab only posts which were made while a mod was active + weak_prior_posts = mod_judged_posts(reddit_submissions) + + # From weak prior posts organize it into successful, + # learning, and other--unclassified + learning_posts = [] + successful_posts = [] + unclassified_posts = [] + for post in weak_prior_posts: + if not isinstance(post.link_flair_text, type(None)): + if 'removed: Learning' == post.link_flair_text: + learning_posts.append(post) + elif post.score >= 6: + successful_posts.append(post) + + else: + unclassified_posts.append(post) + + learning_prior = len(learning_posts)/len(weak_prior_posts) + successful_prior = len(successful_posts)/len(weak_prior_posts) + + learning_S = [post.title for post in learning_posts] + successful_S = [post.title for post in successful_posts] + + learning_word_pairs, learning_pair_count = \ + get_word_pair_probability(learning_S, max_words) + successful_word_pairs, successful_pair_count = \ + get_word_pair_probability(successful_S, max_words) + + + return max_words, learning_prior, successful_prior, \ + learning_word_pairs, learning_pair_count, \ + successful_word_pairs, successful_pair_count + + +class Neg_Log_likehood_Word_Pairs(object): + ''' + This classifier is a weak implementation of a naive bayes classifier + which focuses on the pairs of words used in strings to draw the + classification. To help ensure the struture of the strings are what + the classifier is using, words which do not appear frequently are + replaced with their part of speech. + ''' + def __init__(self, class_prior, word_pair_dict, word_pair_count, allowed_words): + self._clas_prior = class_prior + self._word_pair_dict = word_pair_dict + self._word_pair_count = word_pair_count + self._allowed_words = allowed_words + self._allowed_word_count = len(self._allowed_words) + self._set_missing_word_pair_prob() + def _set_missing_word_pair_prob(self, pos_tag_count=36, modifier=1000): + # https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html + # If it's not in the dataset, it's less than random but not impossible + # the modifier helps keep stuff possible + self._absent_word_pair_prob = (self._allowed_word_count+pos_tag_count)**-2 / modifier + + def neg_log_likelyhood_of_Class_given_words(self, s): + rolling_p = np.log(self._clas_prior) + s = rework_sentence(s, self._allowed_words) + self.any_rep = False # Val indicates if any word pair was in the class database + for i in range(len(s)-1): + pre_word = s[i] # prefix + suf_word = s[i+1] # suffix + p, pair_present = self._get_word_pair_prob(pre_word, suf_word) + if pair_present: + self.any_rep = True + rolling_p += np.log(p) + # if not any_rep: + # print("Nothing to go off of") + #print('-',rolling_p) + return rolling_p + + def _get_word_pair_prob(self, a,b): + D = self._word_pair_dict + pair_count = -1 + rep_present = False + if a in D: + if b in D[a]: + pair_count = D[a][b] + rep_present = True + #print(a,b) + if pair_count == -1: + p = self._absent_word_pair_prob + #print(p) + else: + p = pair_count/ self._word_pair_count + #print(p) + + #print(p) + return p, rep_present + + +class Naive_Bayes_Title_word_pair_classifier(object): + ''' + This is written to have two classes, hard coded, + one is a learning post, class 0, and one is a + successful post class, class 1 + ''' + def __init__(self): + flpath = 'misc/title_classifier.json' + logging.debug("Loading model from "+flpath) + self.load(flname = flpath) + def build(self): + max_words, learning_prior, successful_prior, \ + learning_word_pairs, learning_pair_count, \ + successful_word_pairs, successful_pair_count = build_model() + self.allowed_words = max_words + self._classifiers = [Neg_Log_likehood_Word_Pairs( + class_prior = learning_prior, + word_pair_dict = learning_word_pairs, + word_pair_count = learning_pair_count, + allowed_words = self.allowed_words + ), + Neg_Log_likehood_Word_Pairs( + class_prior = successful_prior, + word_pair_dict = successful_word_pairs, + word_pair_count = successful_pair_count, + allowed_words = self.allowed_words + ) + ] + + def classify_string(self, s): + neg_log_likely = [x.neg_log_likelyhood_of_Class_given_words(s) for x in self._classifiers] + self._s_scores = neg_log_likely[:] + s_class = neg_log_likely.index(max(neg_log_likely)) # Will choose class a in a tie between [a, b] (left most class) + s_score = max(neg_log_likely) + # Grab two highest classes (since there's only two it doesn't matter) + # And build confidence value + del(neg_log_likely[s_class]) + second_score = max(neg_log_likely) + score_diff = s_score - second_score + if score_diff != 0: + s_confidence = np.log(np.abs(score_diff)) + else: + s_confidence = -1000 + + # set string vals to be called on later if needed + self.s_class = s_class + self.s_score = s_score + self.s_confidence = s_confidence + return s_class + + + + + def export(self, ofl_name = 'misc/title_classifier.json', mode='json'): + vals = build_model() + with open(ofl_name, 'w') as f: + json.dump(vals, f) + pass + + def load(self, flname = 'misc/title_classifier.json'): + with open(flname) as json_file: + + max_words, learning_prior, successful_prior, \ + learning_word_pairs, learning_pair_count, \ + successful_word_pairs, successful_pair_count = json.load(json_file) + + self.allowed_words = max_words + self._classifiers = [Neg_Log_likehood_Word_Pairs( + class_prior = learning_prior, + word_pair_dict = learning_word_pairs, + word_pair_count = learning_pair_count, + allowed_words = self.allowed_words + ), + Neg_Log_likehood_Word_Pairs( + class_prior = successful_prior, + word_pair_dict = successful_word_pairs, + word_pair_count = successful_pair_count, + allowed_words = self.allowed_words + ) + ] diff --git a/utils/nb_text_classifier_2.py b/utils/nb_text_classifier_2.py new file mode 100644 index 0000000..c65c5ff --- /dev/null +++ b/utils/nb_text_classifier_2.py @@ -0,0 +1,676 @@ + + + +import numpy as np +import nltk +import datetime +import logging +import json +import re + +import ast +import codeop + + +from aziraphale.data_handlers import redditData +from aziraphale.utils import dataSetTools + + + + +def mod_judged_posts(reddit_submissions): + + ''' + If a post has been removed by a mod, + all posts for the previous 6 hours are + dumped into out_posts on the assumption + that the mod is active, and has allowed the + previous posts to stay up. + ''' + out_posts = [] + posts_buffer = [] + age_limit_hours = 6 + age_limit = datetime.timedelta(hours=age_limit_hours) + learning_post_count = 0 + high_vote_post_count = 0 + + train = [] + test = [] + in_test = False + for post in reddit_submissions: + # Add it to the buffer + if post.created_utc.year >= 2020: + if not in_test: + #print(post.created_utc) + in_test = True + train = out_posts[:] + out_posts = test + posts_buffer.append(post) + youngest_post_dt = post.created_utc + pop_posts = 0 + # Pop old posts + for i in range(len(posts_buffer)): + oldest_post_dt = posts_buffer[i].created_utc + if youngest_post_dt-oldest_post_dt < age_limit: + break + posts_buffer = posts_buffer[i:] + # Check if learning post + if not isinstance(post.link_flair_text, type(None)): + if 'removed: Learning' == post.link_flair_text: + out_posts += posts_buffer + posts_buffer = [] + learning_post_count += 1 + + #print(learning_post_count, len(out_posts)) + test = out_posts + print(len(train), len(test)) + return train, test + +def get_k_most_used_words(samples, k=1000, min_allowed_occurance=2): + ''' + Grab k most used words in the dataset + Anticipates a list of strings + + ''' + word_c = {} + for text in samples: + words = nltk.word_tokenize(text) + words = [x.lower() for x in words] + for word in words: + if word in word_c: + word_c[word] += 1 + else: + word_c[word] = 1 + counts = list(word_c.values()) + words = list(word_c.keys()) + large_indicies = np.array(counts).argsort()[-k:][::-1] + max_words = [words[i] for i in large_indicies] + + return max_words + + +def rework_sentence(s, allowed_words): + try: + t = nltk.word_tokenize(s.lower()) + except TypeError: + print('Tokenize_error') + t = s.lower().split() + pos = nltk.pos_tag(s) + s_out = [] + for i in range(len(t)): + word = t[i] + if word not in allowed_words: + word = pos[i][1].upper() + s_out.append(word) + s_out = ['START'] + s_out + ['END'] + return s_out + + + + +def get_word_pair_probability(S, allowed_words, D=None): + ''' + assumes S is a list of strings + + ''' + if isinstance(D, type(None)): + D = {} + word_pair_count = 0 + for s in S: + s = rework_sentence(s, allowed_words) + word_pair_count += len(s)-1 + for i in range(len(s)-1): + pre_word = s[i] # prefix + suf_word = s[i+1] # suffix + if pre_word in D: + pre_d = D[pre_word] + if suf_word in pre_d: + pre_d[suf_word] += 1 + else: + pre_d[suf_word] = 1 + else: + D[pre_word] = {suf_word:1} + + return D, word_pair_count + + +def unescape_slash(text): + ueex = re.compile(r'(\\)(.)') + #print(ueex.findall(text)) +# out_text = '' +# i = 0 +# n = len(text) +# while True: +# if i >= n: +# break +# if text[i] == '\\': +# i += 1 +# if i < n: +# out_text += text[i] +# i += 1 + ex_text = ueex.sub(r'\2',text) + #if out_text != ex_text: + # print('\t>\t', '\n\t>\t'.join(ex_text.split('\n'))) + return ex_text#out_text + + +def astAndCodeopClassifications(line): + dedent_triggers = ['elif', 'else', 'except'] + dedent_triggers += [x+':' for x in dedent_triggers] + line=line.strip() # Needed to remove whitespace + if line.strip() in ['​', '']: + return 'emptyline' + try: + ast.parse(line.strip()) + c = 'code' + except SyntaxError: + # Possible it's still code, just the opening of a statement + try: + comp = codeop.compile_command(line) + c = 'code' # regardless of comp being none or code object + except SyntaxError: + tokens = line.split() + if tokens[0] in dedent_triggers and line[-1] == ':': + # Need to manually handle dedent case, kinda hand wavey + c = 'code' + else: + c = 'text' + except ValueError: + # Possible output, not sure how it's triggered though + # https://docs.python.org/3/library/codeop.html#codeop.compile_command + c = 'text' + + return c + +def rewrapClassifications(line): + ''' + This is a by hand curriated list of markdown circumstances which + define code blocks. + + This function classifies a line as either empty, codeblock, code, + or NA, + + empty: the line is either a newline character, or the value `​` + + codeblock: A tripple ` is present, starting or ending a codeblock + + code: Either the line leads with at least 4 spaces, or the line starts + and ends with a ` character. + + NA: The line contains text of some form, but it definetly isn't + formatted as code + + ''' + # Function to by hand classify lines + # Should be used to bolster the naive bayes classifier + if line.strip() in ['​', '']: + # 0 width space character, effective newline or standard newline + c = 'emptyline' + elif line.strip()[0:3] == '```': + c = 'codeblock' + elif line.strip()[0] == '`' and line.strip()[-1] == '`': + c = 'code' + elif line[:4] == ' ': + c = 'code' + else: + c = 'NA' + return c +def classifyPostLines(textBlock, classifier): + ''' + This classifier is pretty consistenly all or nothing + A larger code base to train from will help significantly, but is not currently + a critical requirement + ''' + lines = textBlock.split('\n') + classifications = [] + rwclassifications = [] + astClassifieds = [] + for line in lines: + line = line.rstrip() + c = 0#classifier.classify(code_text_features(line)) + classifications.append(c) + rwc = rewrapClassifications(line) + astc = astAndCodeopClassifications(line) + if rwc not in ['code', 'codeblock', 'emptyline'] and astc != 'code': + line = worry_about_autolinks_in_code(line) + if '_LINK_' not in line: + rwc = rewrapClassifications(line) + astc = astAndCodeopClassifications(line) + + rwclassifications.append(rwc) + astClassifieds.append(astc) + #print(c, rwc, line) + ''' + print('\t>',c, line) + dist = classifier.prob_classify(code_text_features(line.strip())) + for label in dist.samples(): + print("\t\t>%s: %f" % (label, dist.prob(label))) + ''' + return classifications, rwclassifications, astClassifieds + +def remove_code_blocks(text): + ''' + Strip out all lines of code for the bot to rework + as [code block] for the selftext naive bayes classifier + + ''' + + + classifier = '' + code_blocked_text = '' + + # Classify the lines in the text + c, rwc, astc = classifyPostLines(text, classifier) + text = text.split('\n') + + line_types = [] + for i in range(len(c)): + # was just `in ['codeblock']` not `in ['code','codeblock']` + # Figure out why and make a comment about it, + if rwc[i] in ['code','codeblock'] or astc[i] == 'code': + line_types.append('c') + elif rwc[i] == 'emptyline': + if i > 0: + line_types.append(line_types[-1]) + else: + line_types.append('t') + else: + line_types.append('t') + + cb = False + for i in range(len(text)): + if line_types[i] == 'c': + if cb == False: + code_blocked_text += '_CODE_\n' + cb = True + else: + code_blocked_text += text[i] + '\n' + cb = False + + + + return code_blocked_text + + +def filter_urls(text): + # Filter out markdown links + mdex = re.compile(r'''\[ # Start of possible link + ([^\]\n]+) # text in link + \]\( # End braket beginning parenthesis + (https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))\S*) # url block + \) # End Url Block + ''', re.VERBOSE) + text = mdex.sub(r'\1', text) + urlex = re.compile(r'https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))\S*') + text = urlex.sub('_LINK_', text) + return text + + +def remove_enum(text): + enex = re.compile(r'[\d](.|:) ') + return enex.sub('* ', text) +def remove_nospace_character(text): + nsex = re.compile(r'​') + return nsex.sub('', text) + +def worry_about_autolinks_in_code(text): + return filter_urls(unescape_slash(text)) + +def hide_inline_code(text): + icex = re.compile(r'`(.*)`') + return icex.sub('_INLINECODE_', text) + +def preprocess_text(text): + text = remove_code_blocks(text) + text = unescape_slash(text) + text = filter_urls(text) + text = remove_nospace_character(text) + text = hide_inline_code(text) + text = remove_enum(text) + return text + + +class Word_Usage_Feature(object): + ''' + A class that'll handle text between submission classes + it'll use n grams so it wont be restricted to words or word pairs + + Given Bayes Equation is $P(A|w_{i}) = /frac{P(A)P(w_{i}|A)}{P(w_{i})}$ + and Given Naive Bayes ignores the normalization via $P(w_{i}$, + this returns the summed log probability of the words given a class + + ''' + def __init__(self, preprocessor, ngram_size=1): + assert ngram_size > 0 + assert type(ngram_size) == int + + self.k = ngram_size + self.member_count = 0 + self.D = {} + + self.text_preprocessor = preprocessor + + # build value for absent ngrams + missing_ngram_modifier = 10*(10**ngram_size) # If it's not in the dataset, + # it's less than random but not impossible + # the modifier helps keep stuff 'possible' + self.missing_ngram = (len(self.text_preprocessor.text_max_words)+\ + self.text_preprocessor.pos_tag_count)**-2 / missing_ngram_modifier + + def add_sample(self, text): + ''' + assuming sents is + assumes space character is not a token + + ''' + if text.strip() == "": + return + sents = self.text_preprocessor.process(text) + for s in sents: + for i in range(len(s)-self.k+1): + token = ' '.join(s[i:i+self.k]) + if token in self.D: + self.D[token] += 1 + else: + self.D[token] = 1 + self.member_count += 1 + return + def get_rolling_p(self, text): + rolling_p = 0 + sents = self.text_preprocessor.process(text) + any_representative = False + for s in sents: + for i in range(len(s)-self.k+1): + token = ' '.join(s[i:i+self.k]) + if token in self.D: + raw_p = self.D[token] / self.member_count + rolling_p += np.log(raw_p) + any_representative = True + else: + rolling_p += np.log(self.missing_ngram) + return rolling_p + + + + + +class simple_feature(object): + def __init__(self, D_Map): + ''' + Dmap is the means to map whatever feature + is provided in add sample to a keyable + value that the dictionary can use + ''' + self.D = {} + self.d_map = D_Map + self.member_count = 0 + + def add_sample(self, s): + key = self.d_map[s] + if key in self.D: + self.D[key] += 1 + else: + self.D[key] = 1 + self.member_count += 1 + def get_rolling_p(self, s): + key = self.d_map[s] + if key in self.D: + return np.log(self.D[key]) + else: + return np.log(1/(self.member_count*100)) # Handles stupid mistakes + + + +class text_preprocessor(object): + def __init__(self, selftext=True, title=False): + self.isselftext = selftext + self.istitle = title + self.remove_uncommon_words = True + self.case_insensative = True + self.n_common_words = 1000 + self.n_added_key_words = 0 + self.pos_tag_count = 36 # https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html + def clean_text(self, text): + text = remove_code_blocks(text) + text = unescape_slash(text) + text = filter_urls(text) + text = remove_nospace_character(text) + text = hide_inline_code(text) + text = remove_enum(text) + return text + def rework_sentence(self,s): + tokens_list, pos_list = self.tokenize_text(s) + s_out = [] + #print(len(tokens_list)) + for sent_index in range(len(tokens_list)): + tokens = tokens_list[sent_index] + pos = pos_list[sent_index] + sent = [] + for i in range(len(tokens)): + word = tokens[i].lower() + if word not in self.text_max_words: + word = pos[i][1].upper() + sent.append(word) + s_out.append(['START'] + sent + ['END']) + + + return s_out + def tokenize_text(self, t): + tokens = [] + pos = [] + if self.isselftext: + text = self.clean_text(t) + lines = text.split('\n') + sents = [] + for s in lines: + sents += nltk.sent_tokenize(s) + elif self.istitle: + sents = nltk.sent_tokenize(t) + for sent in sents: + if len(sent.strip()) > 0: + token_sent = nltk.word_tokenize(sent) + part_of_speech = nltk.pos_tag(token_sent) + if self.case_insensative: + token_sent = [t.lower() for t in token_sent] + tokens.append(token_sent) + pos.append(part_of_speech) + + + return tokens, pos + + def build(self, texts): + self.text_word_count = {} + if self.isselftext: + self.n_added_key_words += 3 + for text in texts: + tokens, pos = self.tokenize_text(text) + sent_count = len(tokens) + for i in range(sent_count): + for token in tokens[i]: + if token in self.text_word_count: + self.text_word_count[token] += 1 + else: + self.text_word_count[token] = 1 + + + counts = list(self.text_word_count.values()) + words = list(self.text_word_count.keys()) + n_max_words = self.n_common_words + self.n_added_key_words + self.text_max_words = [words[i] for i in \ + np.array(counts).argsort()[-n_max_words:][::-1] \ + ] + + + def save(self): + pass + def load(self): + pass + def process(self, text): + text = self.rework_sentence(text) + return text + + +def reddit_link_simplifications(url): + + link_type = "" + if 'i.redd.it' in url: + link_type = 'image' + elif 'https://www.reddit.com/r/Python/' in url: + link_type = 'selftext' + elif '/r/' in url and '/comments/' in url: + link_type = 'xpost' + elif 'v.redd.it' in url: + link_type = 'video' + else: + link_type = 'link' + + return link_type + + + +class reddit_submission_class_fields_naivebayes(object): + ''' + Breaks down and simplifies a class of submissions and builds the + features for + ''' + def __init__(self, p_of_class, title_preprocessor, selftext_preprocessor): + self.p_of_class = p_of_class + self.s =1 + text_ngram = 2 + self.title_word_feat = Word_Usage_Feature(title_preprocessor,ngram_size=text_ngram) + self.resting_karma = simple_feature(2) + self.resting_kratio = simple_feature(2) + self.link_type = simple_feature({'selftext':'selftext', 'link':'link', 'image':'image', 'xpost':'xpost', 'video':'video'}) + self.selftext_word_feat = Word_Usage_Feature(selftext_preprocessor, ngram_size=text_ngram) + + + self.features = {'title':self.title_word_feat, + 'selftext':self.selftext_word_feat, + 'linktype':self.link_type} + + + + def extract_features(self, submission): + title = submission.title + linktype = reddit_link_simplifications(submission.url) + selftext = submission.selftext + + feats = {'title':title, + 'selftext':selftext, + 'linktype':linktype} + return feats + + + def build_features(self, submissions): + for submission in submissions: + feats = self.extract_features(submission) + + for key in feats: + self.features[key].add_sample(feats[key]) + + + def get_p_of_submission_in_class(self, submission): + + log_p = np.log(self.p_of_class) + feats = self.extract_features(submission) + for key in feats: + log_p += self.features[key].get_rolling_p(feats[key]) + + + return log_p + + + +class reddit_submission_classifier(object): + def __init__(self): + pass + def split_classes(self, submissions): + learning_posts = [] + successful_posts = [] + unclassified_posts = [] + for post in submissions: + if not isinstance(post.link_flair_text, type(None)): + if 'removed: Learning' == post.link_flair_text: + learning_posts.append(post) + elif post.score >= 6: + successful_posts.append(post) + + else: + unclassified_posts.append(post) + return learning_posts, successful_posts#, unclassified_posts + + + def mung_submission(self, submission): + ''' + Handle all the feature adjustment that a post + needs to be simply handled by the classifier + ''' + # Adjust to handle 'deleted' and 'removed' messages + if submission.selftext.strip().lower() in ['[deleted]', '[removed]']: + try: + text = submission.edit_History[-1] + if text.strip() != "": + submission.selftext = submission.edit_History[-1] + except: + submission.selftext = "" + + return submission + + def build_classifiers(self, submissions): + self.feature_set = ['common_word_pair_titles', + 'common_word_pair_selftext', + 'post_type'] + submissions = [self.mung_submission(x) for x in submissions] + titles = [x.title for x in submissions] + selftext = [x.selftext for x in submissions] + title_preprocessor = text_preprocessor(selftext=False, title=True) + selftext_preprocessor = text_preprocessor(selftext=True, title=False) + title_preprocessor.build(titles) + selftext_preprocessor.build(selftext) + + source_classes = self.split_classes(submissions) + + self.classes = [] + for aclass in source_classes: + class_p = len(aclass)/len(submissions) + rnb_class = reddit_submission_class_fields_naivebayes(p_of_class=class_p, + title_preprocessor=title_preprocessor, + selftext_preprocessor=selftext_preprocessor) + rnb_class.build_features(aclass) + self.classes.append(rnb_class) + + def classify_submission(self, submission): + scores = [x.get_p_of_submission_in_class(submission) for x in self.classes] + classification = scores.index(max(scores)) + self._scores = scores[:] + self._score = scores[classification] + + + # Get top two classes, and see their sepperation + s_score = max(scores) + del(scores[classification]) + second_score = max(scores) + score_diff = s_score - second_score + if score_diff != 0: + s_confidence = (np.log(np.abs(score_diff)))/(-s_score) + else: + s_confidence = -1000 + self._confidence = s_confidence + + + + return classification + + + +def build_reddit_submission_classifier(): + + handle = redditData.Reddit_Submission_Dataset_Interactions() + reddit_submissions = dataSetTools.DSIterator(handle) + train, test = mod_judged_posts(reddit_submissions) + rsc = reddit_submission_classifier() + + rsc.build_classifiers(train+test) + return rsc + + From fc64d0fa58f3cd08d761b9adbf10016397964f98 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Fri, 17 Jul 2020 14:15:23 -0500 Subject: [PATCH 08/10] Turning off the naive bayes classifier but keeping the code, and addressing broken usernames in the user class --- CHANGELOG.md | 77 ++++++++- FAQ.md | 2 +- README.md | 2 +- ROADMAP.md | 296 +++++++++++++++++++++++++++++++- main.py | 47 ++--- utils/archiveAndUpdateReddit.py | 76 +++----- 6 files changed, 427 insertions(+), 73 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 88712ad..aa9ffc5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ -# CHANGELOG: Python Helper Bot Version pre Alpha A0.4.00 +# CHANGELOG: Python Helper Bot Version pre Alpha A0.4.01 All notable changes to this project will be documented in this file. The format is loosely based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). @@ -18,6 +18,81 @@ Dates follow YYYY-MM-DD format +## [A0.4.01] 2020-07-17 + +Official. + +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +This project is not currently looking for other contributors + +#### Big Picture: What happened, what was worked on +The Naive Bayes classifier is still in the code, but it is currently turned off. + +There are more pressing features, so the code for those functions is frozen until it can be addressed again in the near future. + +#### Added +#### Changed +- `nb_submission_classifier` in main's startup bot function is now `None`. +- `title_classifier` in main now returns false by default and the code which uses `nb_submission_classifier` is commented out. +- `archiveAndUpdatedReddit` now imports praw's `NotFound` exception, and catches it when grabbing a non existent user. +- `phb_Reddit_User` has a attribute `_fake_account` which is set to false unless reddit throws a `NotFound` during lookup. At that point the user is remapped to a collection of fake values. +- `phb_Reddit_User.getUserPosts` now returns and empty list if `_fake_account` +- `phb_Reddit_User.getUserComments` new returns an empty list if `_fake_account` +- Old error listed below `phb_Reddit_user` in a comment has been removed, the above changes now cover it. +- main `getReadyToComment` ignores submissions posted by `User._fake_account` by default as there's no real sense in commenting. + +#### Deprecated +#### Removed + +#### Fixed +#### Security +#### Tests + + + + +### Main +To get this version ready to go, the new nb classifiers are commented out since they're not up to snuff. + +### rpiManager.py + +### Util Libraries + +#### archiveAndUpdateReddit.py +- To get around the issue of users who have an account page that does not exist, the bot populates the user fields with false values, and adds a flag to the user: `_fake_account`. +- The fake account values are empty lists for comments and submissions, account creation time of `datetime.datetime.utcnow()`, an id of `0`, as well as 0 comment and link karma. +#### botHelperFunctions.py +#### botMetrics.py +#### botSummons.py +#### buildComment.py +#### fix_json_archive_bug.py +#### formatBagOfSentences.py +#### formatCode.py +#### locateDB.py +#### lsalib2.py +#### nb_text_classifier.py +#### nb_text_classifier_2.py +#### questionIdentifier.py +#### rpiGPIOFunctions.py +#### scriptedReply.py +#### searchStackOverflowWeb.py +#### startupLoggingCharastics.py +#### summarizeText.py +#### textSupervision.py +#### updateLocalSubHistory.py +#### user_agents.py + +### Tests + + ## [A0.4.00] 2020-07-15 Official. diff --git a/FAQ.md b/FAQ.md index 5303fea..6b5d47c 100644 --- a/FAQ.md +++ b/FAQ.md @@ -134,4 +134,4 @@ When I get to that point, I'll probably just have folks tackle specific elements They seem cool. I've got no problem with them. -#### Version Pre Alpha A0.4.00 \ No newline at end of file +#### Version Pre Alpha A0.4.01 \ No newline at end of file diff --git a/README.md b/README.md index 92fe814..863732b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# Reddit Helper Bot: Version Pre Alpha A0.4.00 +# Reddit Helper Bot: Version Pre Alpha A0.4.01 pythonHelperBot is a reddit bot built to analyze r/python post and determine if they're better suited for the r/learnpython sub. If they are it suggests that the user post to that sub rather than to r/python. diff --git a/ROADMAP.md b/ROADMAP.md index 6f9e3d1..d348591 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ -# ROADMAP: Python Helper Bot Version pre Alpha A0.4.00 +# ROADMAP: Python Helper Bot Version pre Alpha A0.4.01 Future expansions are considered in this file. Their presence is not a promise that they'll exist, but rather this file serves as an early outline of features this project hopes to add, as well as changes in directions @@ -22,6 +22,300 @@ Dates follow YYYY-MM-DD format +## [A0.4.01] 2020-07-17 + +Official. + +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +### Short Term Roadmap + +[X] Turn off naive bayes classifier for the moment to make it easier to + +Once the new classifiers are in place, the bot is going to be reworked again, much more completely this time. +Rather than passing individual classifiers around, there will be a classifier group/class that'll be passed, and it'll handle all individual classifiers. Reddit Submission classes will be incorporated into a larger class which holds the reddit features, as well as that submissions classifications. This may add some ram strain, but the group will make the code easier to follow, debug, and add to. It can be optimized later. + + + +#### Add + - With light archive, build a 'save state' and 'load state' function so records on new posts aren't lost during reboots. This should help resolve 'posts it saw which didn't have help flair, the bot powered down, the flair was applied, and the bot powered up' order of opperation issue. + - botHelperFunctions/botMetrics: Add ram usage check and log it in every 'awake' cycle of program, to try and tease out any possible MemeoryError's kicking up after long periods of time. Might as well save other diagnostics info, maybe call this from the hearbeat thread so it becomes a better representation of runtime + - Full Test Suite: PRIORITY (Ok it'll have to wait until posts have been archived, but it'll still be nice) + - botMetrics.measureUserReaction(): + A function focused on seeing if a user did in fact go to + r/learnpython after the bot made its suggestion. Currently built (kind of, the praw wrappers messed it up a bit), now need to add + functionality in main.py to use it + - Continue Documentation in functions, add documentation files too + - LED Status: A queue which has inputs added to it by functions, and removed by the raspberry pi gpio handler, allowing the bot to change LED status based on what it's doing. Similar to the twitter event bot design (sepperate project). + - botMetrics.predictUserReaction(): A function to go through users comment history, look at the parent comments, and from that gauge how the user will respond to the bots help. In the future adjust how the bot replies based on the predicted responsiveness. For now, it'll just build an archive of users responses to previous comments. + + + +#### Change + - Move NLP functions from NLTK (used in many files) to a buffer module, allowing for simpler, universal changes to be made. For example, if there is a better POS tagger than nltk.pos_tag(sent) then we can easily switch to that. Right now NLTK is used over a fairly large filespace making adjustments of that sort difficult. + - Review all my logging notes. See what should be dropped, changed, etc. + - When errors occur, or a shutdown button is pressed, attempt to save current reddit info into a temp file. on startup, load that info in then connect with praw to expand the knowledge base. + - The archive and update reddit module has become unwieldy. It should be broken into two modules, one which handles the recasting of the praw classes, and one which deals with them as needed.--Ehh.. Maybe not. It does need to be together to a degree + - Migrate away from usage of my personal libraries so it's easier for others to get the bot up and running. + + +#### Deprecate +#### Remove + +#### Fix + - Standardize function name style. Either underscore or camelcase, just not both + Probably preferable to use underscore, the despite camelcase being faster.. + +#### Security +#### Consider + + + +--- + +### General to Long Term Expansion + + - Develop terms for a walk away condition. Either End of active development and the bot remains online, end of active development and death of bot, or end of active development and project is passed on to others. Terms will almost certainly be changed constantly and the project grows and evolves, but it's nice to have an idea of what I consider to be a "complete" project. + + - Numbering system for items in roadmap to clear up what's being worked on and what is completed from an outside perspective. A master numbering system probably is a good idea, vX.X.XX[a,c,d,r,f,s,co]XX, following version, section, and specific roadmap suggestion number. But That seems bloated and unnecessary. (Maybe this isn't worth while, maybe it is and will help catch things in the changelog. Probably wont be seriously considered until alpha) + + - test.EvaluatePost(): + Given recent restructuring, this should be much easier. Take a post given a post id, then run it through the classifier where the exitpoints are turned off from the functions, forcing it to classify the post in full. Because the praw wrappers are in place, there shouldn't be a concern about forcing full evaluation any more. This can be considered to be half completed: the silent mode the bot has helps evaluate posts. + + - Reply To Common posts: + Build semi scripted replies to frequently asked questions (probably largely pulled from the sidebar, since that's how the side bar gets populated) + This includes "How do I install python" and "Is learning python worth it?" (Aka "Why learn python?")--A quality version of this is a full research task. Build off of internal sentence ordering models + soft skills. Pull text from comments on these posts to build up scripted reply. Look into adapting this (or maybe starting with this) to classifying the posts so their types can be diplayed as tags. + - When LDA classifier comes into play, this might be a lot easier. Auto-segment posts into topics, ID topic of question, compare question to similar past question, map past answers to new question. + + - Log processing: a set of functions and visualizations to process the log files for various useful tidbits. Something nicer than grep + + + + +### Main + - alreadyAnswered(): + Parse through OPs comments on the thread, and search for text that implies the question + has been answered. Adjust comment on submission accordingly, probably to say, "Next time + you have a question like this, consider using r/learnpython" blah blah blah + + +### rpiManager.py + - update the commented gpio naming and numbering list at the top of the file + - update grab-from-github functions + - add a queue to work with rpiGPIO for LED displays for various tasks + + + +### Libraries: + + +### archiveAndUpdateReddit.py + + +### botHelperFunctions.py + +### botMetrics.py + - predictUserReaction(): used for the bot gauge how receptive the user will be towards a r/learnpython suggestion. If it predicts combative, it'll make it's comment short and to the point. If it predicts receptive it'll expand on highlighted points such as formatting. If neither it'll contiue as is. --There's probably almost no meat to go off of to help make this prediction though, it's most likely a good idea with no path to implementation. + + - measureUserReaction(): +to see if redditor does post to r/learnpython. The post will have to be strongly similar +to their r/python post, and be posted not long after the python sub post + +- questionAndAnswer(query): to attempt to reply to semi-scripted questions + +- buildConfusionMatrix(): to measure performance + +#### Confusion Matrix Traits +##### True Positive: +[The bot has commented,] And +[[Either a mod has removed the post due to 'learning'], +Or [the redditor posts their question on r/learnpython], Or [the bot has >=2 upvotes]] + +##### False Negative: +[The bot did not comment after 8 hours] And +[[Either a mod has removed the post due to learning,] +Or, [[someone else has commented r/learnpython] and [has greater than 2 upvotes after 8 to 24 +hours after commenting,]] +Or, [the user posted their question on r/learnpython]] + +##### False Positive: +[The bot has commented],And [has less than -1 comment karma after 8 to 24 hours,] +And [[the post is not removed due to learning within 8-24 hours] or [by mods recent +activity plus some threshold.]] +And [[the user does not make a similar post to r/learnpython within a timespan of 8 +hours] or [4 hours after their next user activity monitored for no more than a week]] + +##### True Negative: +[The bot did not comment after 8 hours] And +[[No mod has removed the post using a reference to 'learning' after 8 to 24 hours] or +by mods recent activity plus some threshold.] And +[[No commenter has post made a post which contains 'r/learnpython'] And [has more than 2 upvotes]] + +##### Fuzzy: +All Else. +This class will be either require human moderation to place into the confusion matrix, Or +will be used for other classificaiton, such as "Blog Spam". It could be that topics placed +in this area can be re-examined and labelled, helping the bot generalize preformance in +other areas + +### botSummons.py + - Finish makeFormatHelpMessage summons +### buildComment.py +### formatBagOfSentences.py +### formatCode.py + - formatCode.py: Cleave sentence from comment and first line of code from one another + - formatCode.py: Using rewrapClassifications output, check to see if any indentation is present for lines that have been classified as code. If >5 lines of code are present and none of them have indents, classify block as "The reddit text editor royally screwed this one up", adjust comment to say it's unlikely that the code has been indented properly, and enter the special fixer. + - formatCode.reformatFromHell(): Read in all previous code. Read in current line. If rfh classification Adds indent: current line is a child of the previous line. If it is the same indent level, current line is a sibling. If it is minus indent, line is a sibling of the previous lines parent. + - Previous code is stored in a tree like structure + - Leverage sentence ordering ideology to say given the current line and the previous state of the code tree, which level of node in the tree should I be + This should be an area of linguists where there's plenty of work already completed, look for it. I think Nevil-manning sequitor addresses it briefly, look at that+cited by for other work in the area. + - This might be done with the abstract syntax tree module + + +### learningSubmissionClassifiers.py + +### locateDB.py + - load in path data from a prefernce file, and or take it as input that way the path isn't + 1. Hard coded and + 2. Hard coded in the module + Generic is better if it's generally useful. + + That said, "check_though_these():" is a pretty good and simple function to move out of "locateDB.py" and into main.py + + - Call a function in this library to recast folder/file calls to the correct os format. Or just redo it everywhere in the code. + Whatever works best + +### lsalib2.py +Migrate features back into lsalib + +### nb_text_classifier.py +Needs to be deprecated and removed + +It will need to be cleaned up and swapped out from this frankenstein code and moved to use a more legitimate library. It should also use the same format so the presence of selftext, a link to i.reddit, or a link to a third party site can be added to the calculation, as well as have all of those features added without having to completly rework the core code. + +For selftext posts, consider another weird classification: + break the text into blocks then sentences + As was considered with selftext prior, remap all code to CODE, and merge all neighboring instances of code into one block. + classify each sentence: maybe use LDA to generate m topics, and make a m space. + Final classification for the selftext will be the probility that a question post + built sentences which progressed in that way. This way a rhetorical question is + less likely to mess it up. + + +### nb_text_classifier_2.py +`get_p_of_submission_in_class` needs to use logSumExp trick. + + + + +### questionIdentifier.py +It'd be nice to use stack overflow's user submissions and r/learnpython's +submissions compared to 'successful' r/python submissions to build a 'programmers +question' classifier (and expand the classifier to blogspammers). This would +make it generalizable so posts which are questions or requests ("HELP ME CODE") +are directed to r/learnpython, posts which are clearly for click/ads are commented +on as such, and good posts are 'ignored': allowing redditors to act on it as they +choose. This is not an easy goal to acheive and is incredibly arbitary. but there +are still certain factors which can be measured and acted on. + +This will probably leverage a stack overflow search engine and compare n results +with k or greater similarity. + +### rpiGPIOFunctions.py +### scriptedReply.py +### searchStackOverflowWeb.py + - Scrap and rebuild with approved api and bound it to search for results between + local database build date and present day. Not important until after local copy of SO + is up and running +### startupLoggingCharastics.py +### summarizeText.py + - Improve the english language model for topic modeling, and focus on programming topic modeling. +### textSupervision.py +### updateLocalSubHistory.py +### user_agents.py + - Remove this + +### OTHER +(This is all functions that don't have a clear parent module) + + - moqaProgram + + - ELMO/BERT programs + + - Leverage reformat user code with automatic Q&A: Use classified code regions to match SO code regions, classified text regions to match SO text regions. Hopefully this improves the search engine and cuts the risk of added noise by a text to code block increasing precieved distance between the user query and the SO database post. + Next If a majority of highly matching SO posts have sample code in the question, but the reddit query does not, strongly suggest adding the example code that caused the issue to the next itteration of the query. + + + - question_topic_Modeling(): + This is going to take a few parts. + - Identify all related learning subreddits using a topic model + + - Model the topics of stack overflow questions. + + - Model the topics in the learning subs + Do network analysis to find the most active sub that addresses a topic: probably pagerank since it's simple and it works. It doesn't need to be state of the art, and if it can run on the pi, that's even better + + - Next take in the question, extract topics, feed the topics in the network, identify the sub that will get the best answer fastest. This means there also has to be some knowledge of the subs activity score + + - sub_Activity_Measure(): + Or score.. + This will probably return some arbatrary number that only makes sense in the context of other measures + It might be a function of: + The distance between the top 25 posts on Hot and the top 25 posts in New, where 'top' refers to + reddits ranking. + The number of comments and the absolute value of karma of those comments + the number of unique users in those 25 posts + The time between each activity + + Comparing the intersection of hot to new posts shows a glimps of how active the sub is without requiring the bot to look at the sub at multiple times. + + This function would be useful with the question_topic_modeling() function and wouldn't need to run frequently. Though over multiple runs, it would have a solid understanding of how active a sub is at different times of day, which might encourage the bot to direct a user to a learning sub that is + active at that time. + + + - Auto Reply to common questions (Functional FAQ as it were) + (This is probably going to be an early test of soft skills) + * ["Possibly wanting to learn Python, is it worth it?"](https://www.reddit.com/r/Python/comments/917zxd/) + + - Use Automatic Sentence Ordering to construct the bots autoreply, reducing the mess of the code there. Should be mildly simple (ha, sure...), and allow for much more flexible commenting. Target is to have a defined intro, a 'bag of sentences' for the body, and a defined signature. The 'mildly simple' notion is built off the idea that there will be little the program can do incorrectly with that scaffolding. Look at two metrics: absolute sentence ordering, and new paragraph insertion. Maybe train on a ton of readme's, or wiki data for the new paragraph insertion. + + +#### Question & Answer +Resources to draw from: + - Stack Overflow (Primary) + - Python Docs (Secondary) + - Python Blog Posts (Out of Focus) + - Scraped Github Code (Out of Focus) + +Think about using a subset of highly matching SO posts code to OPs source code and using bayes in a MSAlignment fashion to guess on solution. +Most likely this is especially useful with syntax errors and stack traces. + + +### Generalizing the bot: +These are features which an ideal bot-mod would have, but which are not directly linked to a question-answer-and-redirector bot like u/pythonHelperBot (as of mid July 2018) + - sub_Toxicisty_Score(): + Alternatively a friendly score. Bit ambigous, and doesn't immeadetly fit into the bot, but just a measure of how kind or standoffish or toxic a sub is. Certain communities tend to forget that not everyone knows everything, and it'd be nice to avoid recommending those subs. + + - blog_Spam_Flagger(): + This is actually a large but distant future goal for the bot. There's often complaints about blog spam on the python sub, and it'd be nice to have a programmatic way to define it. Even if the spammy site sees the definition, and works around it, the definition can either be altered, or the work around can be allowed. Most redditors want to see good content, so the best way around an ideal blog spam filter would be to have variable, high quality content. In which case everyone wins. Using that idea, we can start to outline the basic components of what blog spam might be. + + High quality content is safe. High quality with respect to the python sub is probably some function of what generally does well + + Low quality can be caused by a few reasons: + r/python is not the proper sub for that: ie questions + It was recently posted: this is probably best defined as content theft, though repost is a common name for it. + + I'm tired, I'll come back to this. + +### Tests + ## [A0.4.00] 2020-07-15 diff --git a/main.py b/main.py index 2c92890..cbe830a 100644 --- a/main.py +++ b/main.py @@ -230,25 +230,27 @@ def basicQuestion_classifyPost(submission, classifier): def title_classifier(submission, nb_submission_classifier): ''' + CURRENTLY ON PAUSE AND NOT USED + In the current version, learning posts are class 0 successful posts are class 1 ''' - c = nb_submission_classifier.classify_submission(submission) - className = 'successful' if c else 'learning' - logging.debug("Title classified as '"+className.capitalize()+\ - "' With Confidence: "+str(nb_submission_classifier._confidence)+\ - " and with NegLogLikelyhood: "+str(nb_submission_classifier._score)) - print("* ",submission.title) - print("* ",submission.id) - print("* Title classified as '"+className.capitalize()+\ - "' With Confidence: "+str(nb_submission_classifier._confidence)+\ - " and with NegLogLikelyhood: "+str(nb_submission_classifier._score)) - if c == 0 and nb_submission_classifier._confidence > 0.004 and nb_submission_classifier._score > -1000: - logging.debug("Title was strongly classified as learning") - print("* Title was strongly classified as learning") - print("*"*30) - return True - print("*"*30) + # c = nb_submission_classifier.classify_submission(submission) + # className = 'successful' if c else 'learning' + # logging.debug("Title classified as '"+className.capitalize()+\ + # "' With Confidence: "+str(nb_submission_classifier._confidence)+\ + # " and with NegLogLikelyhood: "+str(nb_submission_classifier._score)) + # print("* ",submission.title) + # print("* ",submission.id) + # print("* Title classified as '"+className.capitalize()+\ + # "' With Confidence: "+str(nb_submission_classifier._confidence)+\ + # " and with NegLogLikelyhood: "+str(nb_submission_classifier._score)) + # if c == 0 and nb_submission_classifier._confidence > 0.004 and nb_submission_classifier._score > -1000: + # logging.debug("Title was strongly classified as learning") + # print("* Title was strongly classified as learning") + # print("*"*30) + # return True + # print("*"*30) return False @@ -298,9 +300,12 @@ def getReadyToComment(reddit, setOfPosts, userNames, postHistory, commentOnThese # Check if allowed to comment even if already commented past_interaction = user.name in userNames - buildHelpfulComment(submission, user, reddit, suggested, crossPosted, answered, codePresent, correctlyFormatted, past_interaction, quietMode, phbArcPaths=phbArcPaths) - userNames.append(str(user.name)) - postHistory.append(str(submission.id)) + if not user._fake_account: + buildHelpfulComment(submission, user, reddit, suggested, crossPosted, answered, codePresent, correctlyFormatted, past_interaction, quietMode, phbArcPaths=phbArcPaths) + userNames.append(str(user.name)) + postHistory.append(str(submission.id)) + else: + logging.debug(str(user.name) + " User is a fake deleted or banned account, no sense in commenting") return userNames, postHistory, antiSpamList @@ -332,8 +337,8 @@ def startupBot(): codeVTextClassifier = formatCode.buildTextCodeClassifier(sourceDataPath=paths["codeText"]) # Naive Bayes Title Classifier nb_title_classifier = nb_text_classifier.Naive_Bayes_Title_word_pair_classifier() - nb_submission_classifier = nb_text_classifier_2.build_reddit_submission_classifier() - + #nb_submission_classifier = nb_text_classifier_2.build_reddit_submission_classifier() + nb_submission_classifier = None # Freezing this module for more pressing focuses # Reddit API keySet = getPythonHelperBotKeys.GETREDDIT() diff --git a/utils/archiveAndUpdateReddit.py b/utils/archiveAndUpdateReddit.py index cf75fc1..b794a2f 100644 --- a/utils/archiveAndUpdateReddit.py +++ b/utils/archiveAndUpdateReddit.py @@ -3,7 +3,7 @@ import datetime import praw from praw.exceptions import APIException -from prawcore import RequestException, ResponseException, ServerError +from prawcore import RequestException, ResponseException, ServerError, NotFound from asecretplace import getPythonHelperBotKeys import sqlite3 import json @@ -581,57 +581,12 @@ class phb_Reddit_User(object): ''' - ''' -Error Caused by User Account that was deleted while the bot was looking at/for it. -Rare, but should be handled - -2019-02-28 08:08:49,848 - DEBUG - main.py:getReadyToComment():186 - Processing a valid post -2019-02-28 08:08:49,848 - DEBUG - botHelperFunctions.py:logPostFeatures():54 - ************************************************** -2019-02-28 08:08:49,849 - DEBUG - botHelperFunctions.py:logPostFeatures():55 - [POST] | what is python? -2019-02-28 08:08:49,849 - DEBUG - botHelperFunctions.py:logPostFeatures():56 - [AUTHOR] | shiriyadav -2019-02-28 08:08:49,849 - DEBUG - botHelperFunctions.py:logPostFeatures():57 - [ID] | avnkh8 -2019-02-28 08:08:49,849 - DEBUG - botHelperFunctions.py:logPostFeatures():59 - Post Age: 2:14:43.849659 -2019-02-28 08:08:49,849 - DEBUG - botHelperFunctions.py:logPostFeatures():60 - Votes: 0 -2019-02-28 08:08:49,850 - DEBUG - botHelperFunctions.py:logPostFeatures():61 - Upvote Ratio: 0.1 -2019-02-28 08:08:49,850 - DEBUG - sessions.py:_log_request():49 - Fetching: GET https://oauth.reddit.com/comments/avnkh8/ -2019-02-28 08:08:49,851 - DEBUG - sessions.py:_log_request():50 - Data: None -2019-02-28 08:08:49,851 - DEBUG - sessions.py:_log_request():51 - Params: {'sort': 'best', 'raw_json': 1, 'limit': 2048} -2019-02-28 08:08:50,014 - DEBUG - connectionpool.py:_make_request():393 - https://oauth.reddit.com:443 "GET /comments/avnkh8/?sort=best&raw_json=1&limit=2048 HTTP/1.1" 200 1526 -2019-02-28 08:08:50,018 - DEBUG - sessions.py:_make_request():100 - Response: 200 (1526 bytes) -2019-02-28 08:08:50,024 - DEBUG - sessions.py:_log_request():49 - Fetching: GET https://oauth.reddit.com/user/shiriyadav/submitted -2019-02-28 08:08:50,024 - DEBUG - sessions.py:_log_request():50 - Data: None -2019-02-28 08:08:50,024 - DEBUG - sessions.py:_log_request():51 - Params: {'sort': 'new', 'raw_json': 1, 'limit': 25} -2019-02-28 08:08:50,261 - DEBUG - connectionpool.py:_make_request():393 - https://oauth.reddit.com:443 "GET /user/shiriyadav/submitted?sort=new&raw_json=1&limit=25 HTTP/1.1" 404 38 -2019-02-28 08:08:50,265 - DEBUG - sessions.py:_make_request():100 - Response: 404 (38 bytes) -2019-02-28 08:08:50,265 - ERROR - archiveAndUpdateReddit.py:getUserPosts():618 - Caught Server 500 Error | Specific Error: -2019-02-28 08:08:50,277 - ERROR - archiveAndUpdateReddit.py:getUserPosts():619 - -Traceback (most recent call last): - File "/home/pi/Documents/filesForProgramming/Reddit/pythonHelpBot2/utils/archiveAndUpdateReddit.py", line 593, in getUserPosts - for submission in praw_user.submissions.new(limit=limitCount): - File "/usr/local/lib/python2.7/dist-packages/praw/models/listing/generator.py", line 80, in next - return self.__next__() - File "/usr/local/lib/python2.7/dist-packages/praw/models/listing/generator.py", line 52, in __next__ - self._next_batch() - File "/usr/local/lib/python2.7/dist-packages/praw/models/listing/generator.py", line 62, in _next_batch - self._listing = self._reddit.get(self.url, params=self.params) - File "/usr/local/lib/python2.7/dist-packages/praw/reddit.py", line 371, in get - data = self.request('GET', path, params=params) - File "/usr/local/lib/python2.7/dist-packages/praw/reddit.py", line 486, in request - params=params) - File "/usr/local/lib/python2.7/dist-packages/prawcore/sessions.py", line 182, in request - params=params, url=url) - File "/usr/local/lib/python2.7/dist-packages/prawcore/sessions.py", line 127, in _request_with_retries - raise self.STATUS_EXCEPTIONS[response.status_code](response) -NotFound: received 404 HTTP response - - - - ''' def __init__(self, praw_user): vals_Assigned = False self.directed_to_learning_sub = False self.said_thanks_or_it_worked = False + self._fake_account = False # Prep for errors maxTotalWaitTime = 5*60*60 @@ -656,6 +611,20 @@ def __init__(self, praw_user): self.comment_karma = praw_user.comment_karma vals_Assigned = True break + except NotFound: + ''' + Username does not exist or isn't accessible + set fake account to true + populate fake values + ''' + logging.warning("User " + self.name + " Does not exist. Populating with fake values") + self._fake_account = True + self.created_utc = now + self.id = 0 + self.link_karma = 0 + self.comment_karma = 0 + vals_Assigned = True + except RequestException as e: logging.error("Caught Server Rate Limit Hit | Specific Error:") logging.error("\n"+traceback.format_exc()) @@ -716,6 +685,10 @@ def getUserPosts(self, reddit, limitCount=25, ageLimitHours=None, ageLimitTime=N maxBackoffTime = 5*60 startTime = time.time() + if self._fake_account: + submissionList = [] + return submissionList + while True: if vals_Assigned: break @@ -786,6 +759,10 @@ def getUsersComments(self, reddit, limitCount=25, ageLimitHours=None, ageLimitTi serverBackoffTime = 5 maxBackoffTime = 5*60 startTime = time.time() + + if self._fake_account: + commentList = [] + return commentList while True: if vals_Assigned: @@ -1471,9 +1448,12 @@ def getMods(reddit, sub="python"): if vals_Assigned: break try: + self_name = reddit.user.me().name mods = [] for mod in reddit.subreddit(sub).moderator(): - mods.append(phb_Reddit_User(mod)) + # Ignore bots actions + if mod != self_name: + mods.append(phb_Reddit_User(mod)) vals_Assigned = True break except RequestException as e: From a8ca5636c67ec4831aa2bd0948cb23e4c1195e88 Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Fri, 17 Jul 2020 14:27:56 -0500 Subject: [PATCH 09/10] Begin preping v0.5.00 documentation --- CHANGELOG.md | 63 ++++++++++- FAQ.md | 2 +- README.md | 2 +- ROADMAP.md | 297 ++++++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 360 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa9ffc5..2c5ff83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ -# CHANGELOG: Python Helper Bot Version pre Alpha A0.4.01 +# CHANGELOG: Python Helper Bot Version pre Alpha A0.5.00 All notable changes to this project will be documented in this file. The format is loosely based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). @@ -17,6 +17,67 @@ Dates follow YYYY-MM-DD format -- Susan Calvin in "I, Robot" by Isaac Asimov +## [A0.5.00] 2020-07-XX + +Official. + +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +This project is not currently looking for other contributors + +#### Big Picture: What happened, what was worked on + +Major redesign and beginning new roles + +#### Added +#### Changed +#### Deprecated +#### Removed +#### Fixed +#### Security +#### Tests + + + + +### Main + +### rpiManager.py + +### Util Libraries + +#### archiveAndUpdateReddit.py +#### botHelperFunctions.py +#### botMetrics.py +#### botSummons.py +#### buildComment.py +#### fix_json_archive_bug.py +#### formatBagOfSentences.py +#### formatCode.py +#### locateDB.py +#### lsalib2.py +#### nb_text_classifier.py +#### nb_text_classifier_2.py +#### questionIdentifier.py +#### rpiGPIOFunctions.py +#### scriptedReply.py +#### searchStackOverflowWeb.py +#### startupLoggingCharastics.py +#### summarizeText.py +#### textSupervision.py +#### updateLocalSubHistory.py +#### user_agents.py + +### Tests + + ## [A0.4.01] 2020-07-17 diff --git a/FAQ.md b/FAQ.md index 6b5d47c..0e3d45e 100644 --- a/FAQ.md +++ b/FAQ.md @@ -134,4 +134,4 @@ When I get to that point, I'll probably just have folks tackle specific elements They seem cool. I've got no problem with them. -#### Version Pre Alpha A0.4.01 \ No newline at end of file +#### Version Pre Alpha A0.5.00 \ No newline at end of file diff --git a/README.md b/README.md index 863732b..3d25de7 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# Reddit Helper Bot: Version Pre Alpha A0.4.01 +# Reddit Helper Bot: Version Pre Alpha A0.5.00 pythonHelperBot is a reddit bot built to analyze r/python post and determine if they're better suited for the r/learnpython sub. If they are it suggests that the user post to that sub rather than to r/python. diff --git a/ROADMAP.md b/ROADMAP.md index d348591..e6e9e44 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ -# ROADMAP: Python Helper Bot Version pre Alpha A0.4.01 +# ROADMAP: Python Helper Bot Version pre Alpha A0.5.00 Future expansions are considered in this file. Their presence is not a promise that they'll exist, but rather this file serves as an early outline of features this project hopes to add, as well as changes in directions @@ -21,6 +21,301 @@ Dates follow YYYY-MM-DD format -- Susan Calvin in "I, Robot" by Isaac Asimov +## [A0.5.00] 2020-07-XX + +Official. + +### Contributors +Keith Murray + +email: kmurrayis@gmail.com | +twitter: [@keithTheEE](https://twitter.com/keithTheEE) | +github: [CrakeNotSnowman](https://github.com/CrakeNotSnowman) + +Unless otherwise noted, all changes by @kmurrayis + +### Short Term Roadmap + +[X] Turn off naive bayes classifier for the moment to make it easier to + +Once the new classifiers are in place, the bot is going to be reworked again, much more completely this time. +Rather than passing individual classifiers around, there will be a classifier group/class that'll be passed, and it'll handle all individual classifiers. Reddit Submission classes will be incorporated into a larger class which holds the reddit features, as well as that submissions classifications. This may add some ram strain, but the group will make the code easier to follow, debug, and add to. It can be optimized later. + + + +#### Add + - With light archive, build a 'save state' and 'load state' function so records on new posts aren't lost during reboots. This should help resolve 'posts it saw which didn't have help flair, the bot powered down, the flair was applied, and the bot powered up' order of opperation issue. + - botHelperFunctions/botMetrics: Add ram usage check and log it in every 'awake' cycle of program, to try and tease out any possible MemeoryError's kicking up after long periods of time. Might as well save other diagnostics info, maybe call this from the hearbeat thread so it becomes a better representation of runtime + - Full Test Suite: PRIORITY (Ok it'll have to wait until posts have been archived, but it'll still be nice) + - botMetrics.measureUserReaction(): + A function focused on seeing if a user did in fact go to + r/learnpython after the bot made its suggestion. Currently built (kind of, the praw wrappers messed it up a bit), now need to add + functionality in main.py to use it + - Continue Documentation in functions, add documentation files too + - LED Status: A queue which has inputs added to it by functions, and removed by the raspberry pi gpio handler, allowing the bot to change LED status based on what it's doing. Similar to the twitter event bot design (sepperate project). + - botMetrics.predictUserReaction(): A function to go through users comment history, look at the parent comments, and from that gauge how the user will respond to the bots help. In the future adjust how the bot replies based on the predicted responsiveness. For now, it'll just build an archive of users responses to previous comments. + + + +#### Change + - Move NLP functions from NLTK (used in many files) to a buffer module, allowing for simpler, universal changes to be made. For example, if there is a better POS tagger than nltk.pos_tag(sent) then we can easily switch to that. Right now NLTK is used over a fairly large filespace making adjustments of that sort difficult. + - Review all my logging notes. See what should be dropped, changed, etc. + - When errors occur, or a shutdown button is pressed, attempt to save current reddit info into a temp file. on startup, load that info in then connect with praw to expand the knowledge base. + - The archive and update reddit module has become unwieldy. It should be broken into two modules, one which handles the recasting of the praw classes, and one which deals with them as needed.--Ehh.. Maybe not. It does need to be together to a degree + - Migrate away from usage of my personal libraries so it's easier for others to get the bot up and running. + + +#### Deprecate +#### Remove + +#### Fix + - Standardize function name style. Either underscore or camelcase, just not both + Probably preferable to use underscore, the despite camelcase being faster.. + +#### Security +#### Consider + + + +--- + +### General to Long Term Expansion + + - Develop terms for a walk away condition. Either End of active development and the bot remains online, end of active development and death of bot, or end of active development and project is passed on to others. Terms will almost certainly be changed constantly and the project grows and evolves, but it's nice to have an idea of what I consider to be a "complete" project. + + - Numbering system for items in roadmap to clear up what's being worked on and what is completed from an outside perspective. A master numbering system probably is a good idea, vX.X.XX[a,c,d,r,f,s,co]XX, following version, section, and specific roadmap suggestion number. But That seems bloated and unnecessary. (Maybe this isn't worth while, maybe it is and will help catch things in the changelog. Probably wont be seriously considered until alpha) + + - test.EvaluatePost(): + Given recent restructuring, this should be much easier. Take a post given a post id, then run it through the classifier where the exitpoints are turned off from the functions, forcing it to classify the post in full. Because the praw wrappers are in place, there shouldn't be a concern about forcing full evaluation any more. This can be considered to be half completed: the silent mode the bot has helps evaluate posts. + + - Reply To Common posts: + Build semi scripted replies to frequently asked questions (probably largely pulled from the sidebar, since that's how the side bar gets populated) + This includes "How do I install python" and "Is learning python worth it?" (Aka "Why learn python?")--A quality version of this is a full research task. Build off of internal sentence ordering models + soft skills. Pull text from comments on these posts to build up scripted reply. Look into adapting this (or maybe starting with this) to classifying the posts so their types can be diplayed as tags. + - When LDA classifier comes into play, this might be a lot easier. Auto-segment posts into topics, ID topic of question, compare question to similar past question, map past answers to new question. + + - Log processing: a set of functions and visualizations to process the log files for various useful tidbits. Something nicer than grep + + + + +### Main + - alreadyAnswered(): + Parse through OPs comments on the thread, and search for text that implies the question + has been answered. Adjust comment on submission accordingly, probably to say, "Next time + you have a question like this, consider using r/learnpython" blah blah blah + + +### rpiManager.py + - update the commented gpio naming and numbering list at the top of the file + - update grab-from-github functions + - add a queue to work with rpiGPIO for LED displays for various tasks + + + +### Libraries: + + +### archiveAndUpdateReddit.py + + +### botHelperFunctions.py + +### botMetrics.py + - predictUserReaction(): used for the bot gauge how receptive the user will be towards a r/learnpython suggestion. If it predicts combative, it'll make it's comment short and to the point. If it predicts receptive it'll expand on highlighted points such as formatting. If neither it'll contiue as is. --There's probably almost no meat to go off of to help make this prediction though, it's most likely a good idea with no path to implementation. + + - measureUserReaction(): +to see if redditor does post to r/learnpython. The post will have to be strongly similar +to their r/python post, and be posted not long after the python sub post + +- questionAndAnswer(query): to attempt to reply to semi-scripted questions + +- buildConfusionMatrix(): to measure performance + +#### Confusion Matrix Traits +##### True Positive: +[The bot has commented,] And +[[Either a mod has removed the post due to 'learning'], +Or [the redditor posts their question on r/learnpython], Or [the bot has >=2 upvotes]] + +##### False Negative: +[The bot did not comment after 8 hours] And +[[Either a mod has removed the post due to learning,] +Or, [[someone else has commented r/learnpython] and [has greater than 2 upvotes after 8 to 24 +hours after commenting,]] +Or, [the user posted their question on r/learnpython]] + +##### False Positive: +[The bot has commented],And [has less than -1 comment karma after 8 to 24 hours,] +And [[the post is not removed due to learning within 8-24 hours] or [by mods recent +activity plus some threshold.]] +And [[the user does not make a similar post to r/learnpython within a timespan of 8 +hours] or [4 hours after their next user activity monitored for no more than a week]] + +##### True Negative: +[The bot did not comment after 8 hours] And +[[No mod has removed the post using a reference to 'learning' after 8 to 24 hours] or +by mods recent activity plus some threshold.] And +[[No commenter has post made a post which contains 'r/learnpython'] And [has more than 2 upvotes]] + +##### Fuzzy: +All Else. +This class will be either require human moderation to place into the confusion matrix, Or +will be used for other classificaiton, such as "Blog Spam". It could be that topics placed +in this area can be re-examined and labelled, helping the bot generalize preformance in +other areas + +### botSummons.py + - Finish makeFormatHelpMessage summons +### buildComment.py +### formatBagOfSentences.py +### formatCode.py + - formatCode.py: Cleave sentence from comment and first line of code from one another + - formatCode.py: Using rewrapClassifications output, check to see if any indentation is present for lines that have been classified as code. If >5 lines of code are present and none of them have indents, classify block as "The reddit text editor royally screwed this one up", adjust comment to say it's unlikely that the code has been indented properly, and enter the special fixer. + - formatCode.reformatFromHell(): Read in all previous code. Read in current line. If rfh classification Adds indent: current line is a child of the previous line. If it is the same indent level, current line is a sibling. If it is minus indent, line is a sibling of the previous lines parent. + - Previous code is stored in a tree like structure + - Leverage sentence ordering ideology to say given the current line and the previous state of the code tree, which level of node in the tree should I be + This should be an area of linguists where there's plenty of work already completed, look for it. I think Nevil-manning sequitor addresses it briefly, look at that+cited by for other work in the area. + - This might be done with the abstract syntax tree module + + +### learningSubmissionClassifiers.py + +### locateDB.py + - load in path data from a prefernce file, and or take it as input that way the path isn't + 1. Hard coded and + 2. Hard coded in the module + Generic is better if it's generally useful. + + That said, "check_though_these():" is a pretty good and simple function to move out of "locateDB.py" and into main.py + + - Call a function in this library to recast folder/file calls to the correct os format. Or just redo it everywhere in the code. + Whatever works best + +### lsalib2.py +Migrate features back into lsalib + +### nb_text_classifier.py +Needs to be deprecated and removed + +It will need to be cleaned up and swapped out from this frankenstein code and moved to use a more legitimate library. It should also use the same format so the presence of selftext, a link to i.reddit, or a link to a third party site can be added to the calculation, as well as have all of those features added without having to completly rework the core code. + +For selftext posts, consider another weird classification: + break the text into blocks then sentences + As was considered with selftext prior, remap all code to CODE, and merge all neighboring instances of code into one block. + classify each sentence: maybe use LDA to generate m topics, and make a m space. + Final classification for the selftext will be the probility that a question post + built sentences which progressed in that way. This way a rhetorical question is + less likely to mess it up. + + +### nb_text_classifier_2.py +`get_p_of_submission_in_class` needs to use logSumExp trick. + + + + +### questionIdentifier.py +It'd be nice to use stack overflow's user submissions and r/learnpython's +submissions compared to 'successful' r/python submissions to build a 'programmers +question' classifier (and expand the classifier to blogspammers). This would +make it generalizable so posts which are questions or requests ("HELP ME CODE") +are directed to r/learnpython, posts which are clearly for click/ads are commented +on as such, and good posts are 'ignored': allowing redditors to act on it as they +choose. This is not an easy goal to acheive and is incredibly arbitary. but there +are still certain factors which can be measured and acted on. + +This will probably leverage a stack overflow search engine and compare n results +with k or greater similarity. + +### rpiGPIOFunctions.py +### scriptedReply.py +### searchStackOverflowWeb.py + - Scrap and rebuild with approved api and bound it to search for results between + local database build date and present day. Not important until after local copy of SO + is up and running +### startupLoggingCharastics.py +### summarizeText.py + - Improve the english language model for topic modeling, and focus on programming topic modeling. +### textSupervision.py +### updateLocalSubHistory.py +### user_agents.py + - Remove this + +### OTHER +(This is all functions that don't have a clear parent module) + + - moqaProgram + + - ELMO/BERT programs + + - Leverage reformat user code with automatic Q&A: Use classified code regions to match SO code regions, classified text regions to match SO text regions. Hopefully this improves the search engine and cuts the risk of added noise by a text to code block increasing precieved distance between the user query and the SO database post. + Next If a majority of highly matching SO posts have sample code in the question, but the reddit query does not, strongly suggest adding the example code that caused the issue to the next itteration of the query. + + + - question_topic_Modeling(): + This is going to take a few parts. + - Identify all related learning subreddits using a topic model + + - Model the topics of stack overflow questions. + + - Model the topics in the learning subs + Do network analysis to find the most active sub that addresses a topic: probably pagerank since it's simple and it works. It doesn't need to be state of the art, and if it can run on the pi, that's even better + + - Next take in the question, extract topics, feed the topics in the network, identify the sub that will get the best answer fastest. This means there also has to be some knowledge of the subs activity score + + - sub_Activity_Measure(): + Or score.. + This will probably return some arbatrary number that only makes sense in the context of other measures + It might be a function of: + The distance between the top 25 posts on Hot and the top 25 posts in New, where 'top' refers to + reddits ranking. + The number of comments and the absolute value of karma of those comments + the number of unique users in those 25 posts + The time between each activity + + Comparing the intersection of hot to new posts shows a glimps of how active the sub is without requiring the bot to look at the sub at multiple times. + + This function would be useful with the question_topic_modeling() function and wouldn't need to run frequently. Though over multiple runs, it would have a solid understanding of how active a sub is at different times of day, which might encourage the bot to direct a user to a learning sub that is + active at that time. + + + - Auto Reply to common questions (Functional FAQ as it were) + (This is probably going to be an early test of soft skills) + * ["Possibly wanting to learn Python, is it worth it?"](https://www.reddit.com/r/Python/comments/917zxd/) + + - Use Automatic Sentence Ordering to construct the bots autoreply, reducing the mess of the code there. Should be mildly simple (ha, sure...), and allow for much more flexible commenting. Target is to have a defined intro, a 'bag of sentences' for the body, and a defined signature. The 'mildly simple' notion is built off the idea that there will be little the program can do incorrectly with that scaffolding. Look at two metrics: absolute sentence ordering, and new paragraph insertion. Maybe train on a ton of readme's, or wiki data for the new paragraph insertion. + + +#### Question & Answer +Resources to draw from: + - Stack Overflow (Primary) + - Python Docs (Secondary) + - Python Blog Posts (Out of Focus) + - Scraped Github Code (Out of Focus) + +Think about using a subset of highly matching SO posts code to OPs source code and using bayes in a MSAlignment fashion to guess on solution. +Most likely this is especially useful with syntax errors and stack traces. + + +### Generalizing the bot: +These are features which an ideal bot-mod would have, but which are not directly linked to a question-answer-and-redirector bot like u/pythonHelperBot (as of mid July 2018) + - sub_Toxicisty_Score(): + Alternatively a friendly score. Bit ambigous, and doesn't immeadetly fit into the bot, but just a measure of how kind or standoffish or toxic a sub is. Certain communities tend to forget that not everyone knows everything, and it'd be nice to avoid recommending those subs. + + - blog_Spam_Flagger(): + This is actually a large but distant future goal for the bot. There's often complaints about blog spam on the python sub, and it'd be nice to have a programmatic way to define it. Even if the spammy site sees the definition, and works around it, the definition can either be altered, or the work around can be allowed. Most redditors want to see good content, so the best way around an ideal blog spam filter would be to have variable, high quality content. In which case everyone wins. Using that idea, we can start to outline the basic components of what blog spam might be. + + High quality content is safe. High quality with respect to the python sub is probably some function of what generally does well + + Low quality can be caused by a few reasons: + r/python is not the proper sub for that: ie questions + It was recently posted: this is probably best defined as content theft, though repost is a common name for it. + + I'm tired, I'll come back to this. + +### Tests + + ## [A0.4.01] 2020-07-17 From 37cacb70fbc3d64a93c04c305a9d2cf820680ffd Mon Sep 17 00:00:00 2001 From: CrakeNotSnowman Date: Fri, 17 Jul 2020 15:02:58 -0500 Subject: [PATCH 10/10] Swapped the completed note I missed with 'in progress' --- CHANGELOG.md | 2 +- ROADMAP.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c5ff83..ab5995d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ Dates follow YYYY-MM-DD format ## [A0.5.00] 2020-07-XX -Official. +In Progress ### Contributors Keith Murray diff --git a/ROADMAP.md b/ROADMAP.md index e6e9e44..e5789bd 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -23,7 +23,7 @@ Dates follow YYYY-MM-DD format ## [A0.5.00] 2020-07-XX -Official. +In Progress ### Contributors Keith Murray