Re: Regarding Text to HTML conversion

Thanks for offering Brent!

Overall goal: reduce the tedium of minute generation after the calls. It's
a burden Manu and team had been carrying for a long time, and then I
misguidedly volunteered to take over. :) I think it takes about 20-30
minutes each week -- just enough time to be a pain given other obligations.

I documented the process here (https://w3c-ccg.github.io/publish.html) --
also note the link to the video demonstration. Recently I've been thinking
about ways to automate things. For example, at first I assumed the need to
edit the audio was a given, and so I started going down a path (auto-wave
detection and trimming) that made me suspicious I needed to learn the
bigger picture. The voipbot code and asterisk libraries addressed the gap
in my understanding, and now I see what sort of changes we might make.

The tedious parts I see are:
*1. Trimming dead audio*
The need to trim dead audio from beginning and end. We do this manually in
an audio editor. The need to even install such an app is (IMO) a source of
friction that prevents people from even wanting to help.

Fortunately there's a better solution. Asterisk exposes "start"/"stop"
recording commands. I'm thinking we could add a voipbot command so that the
chair can start recording (or if we're worried that's something we'll
forget, we could tie it to another trigger that we use around the start of
the meeting.

*2. Minute cleanup*
I don't have a good proposal for this yet. This part involves editing the
irc.log text files. The instructions to do this (currently) are:

   - Go to the online scribe tool and copy/paste irc.log into the text
   input box at the bottom.
   - Clean up the IRC log accordingly and overwrite irc.log with the edited
   file.
   - Things to check
      - Look for any find/replace suggestions in irc.log and update them
      (s/../..)
      - Ensure (people name) aliases have matches (there is a people.json
      file in the publishing repo)

But there's another problematic part, which is ensuring everyone on the
call has been accounting for. This is currently an error-prone step because
it's up to the publisher to (informally) remember to which people joined in
and add them.

Progress on 1 or 2 would be great. Manu, I'm curious if you have any
objections to explicit start/stop recording -- perhaps there's a historical
reason for that.

As for the existing minutes, I think we have no choice but to suck it up
and publish them using the existing means. But working on these
improvements for future recordings would tremendously helpful.

Thanks,
Kim




On Fri, Aug 9, 2019 at 2:48 PM Brent Shambaugh <brent.shambaugh@gmail.com>
wrote:

> Is there some way I can help? What is the goal?
>
> I do see that
> https://github.com/digitalbazaar/voipbot/blob/master/index.js has a lot
> of comments in it already. This could be the first place to look.
>
> Running grep "//[A-Za-z0-9]" on this gives:
>
> // read the config file
> // create a lockfile that goes stale after 130 minutes
> // cleanup the lockfile if there is an uncaught exception or an interrupt
> // shared variables
> // check to see if there is a channel password
> // connect to the IRC channel
> // check if TLS should be used to connect
> // check to see if the server requires a password
> // says the given message in the main irc channel
> // hook up an error handler to prevent exit on error
> // handle channel join event
>   // connect to the Asterisk server
>   // get a list of conference participants on join
>   // announce when people join the conference
>   // confbridge talking the recording file
>       // current time in seconds since epoch
>       // clear audio events older than 3 minutes
>   // build the list of participants
>   // announce when people leave the conference
>   // listen to IRC channel messages
>     // log IRC messages to logfile if a recording directory is specified
>       // log, ignoring all errors
>     // handle non-voipbot directed channel commands
>     // handle voipbot specific commands
>     // all commands must contain at least the command name which is
>     // the second argument
>     // show list of participants
>       // current time in seconds since epoch
>       // overwrite calleridname
>   // must have at least two characters to attempt a guess
>   // guess by attempting to match the end of the channel name
>     // upload log files if they exist
> // search a directory for a file matching regexp larger than fsize bytes
>   // return early if upload settings are not set
>   // get latest audio file larger than 15MB
>   // get latest audio and IRC logs
>     s3ForcePathStyle: true, // needed with minio?
>       // build S3 parameters
>       // build S3 parameters
>
> other:
>
> /*************************** Helper functions ******************************/
>
> /**
>  * Converts a queue to a string.
>  *
>  * @param queue the queue to convert to a string
>  * @return a string representing the queue
>  */
>
> /**
>  * Removes a given nick from a queue.
>  *
>  * @param queue the queue to modify.
>  * @param nick the nickname to remove from the queue
>  *
>  * @return the removed value.
>  */
>
> /**
>  * Pretty-prints a channel name for human readability.
>  *
>  * @param channel the channel name
>  * @return the pretty-printed channel
>  */
>
> /**
>  * Attempts to guess a channel given some text. The test currently tries
>  * to match the last section of the channel name.
>  *
>  * @param text the text to try and match against the end of the channel name.
>  * @return the guessed channel name, or false if the guess failed.
>  */
>
> /**
>  * Removes a given nick from a queue.
>  *
>  * @param queue the queue to modify.
>  * @param nick the nickname to remove from the queue
>  *
>  * @return the removed value.
>  */
>
> // search a directory for a file matching regexp larger than fsize bytes
>
>
> It looks like .wav files are handled in the program. I see no mention of
> .mp4. I guess the conversion program from .wav to .mp4 elsewhere?
> It looks like the script works with https://aws.amazon.com/s3/ .
>
> At least this documentation might be functionally equivalent.
> https://www.voip-info.org/asterisk-manager-api/
> The wiki might be useful, but it is difficult to find a starting point.
> https://wiki.asterisk.org/wiki/display/AST/Beginning+Asterisk
>
> -Brent Shambaugh
>
> GitHub: https://github.com/bshambaugh
> Website: http://bshambaugh.org/
> LinkedIN: https://www.linkedin.com/in/brent-shambaugh-9b91259
> Skype: brent.shambaugh
> Twitter: https://twitter.com/Brent_Shambaugh
> WebID: http://bshambaugh.org/foaf.rdf#me
>
>
> On Fri, Aug 9, 2019 at 2:08 PM Kim Hamilton <kimdhamilton@gmail.com>
> wrote:
>
>> Excellent thanks! I’ll take a look
>>
>> On Fri, Aug 9, 2019 at 10:09 AM Manu Sporny <msporny@digitalbazaar.com>
>> wrote:
>>
>>> On 8/9/19 1:05 AM, Kim Hamilton wrote:
>>> > Where is the info/source for voipbot?
>>>
>>> https://github.com/digitalbazaar/voipbot
>>>
>>> Documentation is horrible, integration into Asterisk is beyond
>>> painful... it's a volunteer project built over the last decade in our
>>> spare time. :)
>>>
>>> That said, it's the heart and soul of the system that manages the phone
>>> lines, audio recording, log publishing, and queue management for our
>>> weekly CCG calls. We'd be happy to deploy upgrades if anyone would like
>>> to contribute new features, bug fixes, etc.
>>>
>>> -- manu
>>>
>>> --
>>> Manu Sporny (skype: msporny, twitter: manusporny)
>>> Founder/CEO - Digital Bazaar, Inc.
>>> blog: Veres One Decentralized Identifier Blockchain Launches
>>> https://tinyurl.com/veres-one-launches
>>>
>>>