nvchecker/README.rst

**nvchecker** (short for *new version checker*) is for checking if a new version of some software has been released.

.. image:: https://travis-ci.org/lilydjwg/nvchecker.svg
   :alt: Build Status
   :target: https://travis-ci.org/lilydjwg/nvchecker
.. image:: https://badge.fury.io/py/nvchecker.svg
   :alt: PyPI version
   :target: https://badge.fury.io/py/nvchecker

Contents
========

* `Dependency <#dependency>`_
* `Install and Run <#running>`_

  * `JSON logging <#json-logging>`_

* `Version Record Files <#version-record-files>`_

  * `The nvtake Command <#the-nvtake-command>`_

* `Version Source Files <#version-source-files>`_

  * `Configuration Section <#configuration-section>`_
  * `Global Optons <#global-options>`_
  * `Search in a Webpage <#search-in-a-webpage>`_
  * `Find with a Command <#find-with-a-command>`_
  * `Check AUR <#check-aur>`_
  * `Check GitHub <#check-github>`_
  * `Check BitBucket <#check-bitbucket>`_
  * `Check GitLab <#check-gitlab>`_
  * `Check PyPI <#check-pypi>`_
  * `Check RubyGems <#check-rubygems>`_
  * `Check NPM Registry <#check-npm-registry>`_
  * `Check Hackage <#check-hackage>`_
  * `Check CPAN <#check-cpan>`_
  * `Check Packagist <#check-packagist>`_
  * `Check Local Pacman Database <#check-local-pacman-database>`_
  * `Check Arch Linux official packages <#check-arch-linux-official-packages>`_
  * `Check Debian Linux official packages <#check-debian-linux-official-packages>`_
  * `Check Ubuntu Linux official packages <#check-ubuntu-linux-official-packages>`_
  * `Check Anitya (release-monitoring.org) <#check-anitya>`_
  * `Check Android SDK <#check-android-sdk>`_
  * `Manually updating <#manually-updating>`_
  * `Version Control System (VCS) (git, hg, svn, bzr) <#version-control-system-vcs-git-hg-svn-bzr>`_
  * `Other <#other>`_

* `Bugs <#bugs>`_
* `Footnotes <#footnotes>`_

Dependency
==========
- Python 3.5+
- Python library: structlog
- One of these Python library combinations (ordered by preference):

  * tornado + pycurl
  * aiohttp
  * tornado

- All commands used in your version source files

Install and Run
===============
To install::

  pip3 install nvchecker

To use the latest code, you can also clone this repository and run::

  python3 setup.py install

To see available options::

  nvchecker --help

Run with one or more software version source files::

  nvchecker source_file

You normally will like to specify some "version record files"; see below.

JSON logging
------------
With ``--logger=json`` or ``--logger=both``, you can get a structured logging
for programmatically consuming. You can use ``--json-log-fd=N`` to specify the
file descriptor to send logs to (take care to do line buffering). The logging
level option (``-l`` or ``--logging``) doesn't take effect with this.

The JSON log is one JSON string per line. The following documented events and
fields are stable, undocumented ones may change without notice.

event=updated
  An update is detected. Fields ``name``, ``old_version`` and ``version`` are
  available. ``old_version`` maybe ``null``.

event=up-to-date
  There is no update. Fields ``name`` and ``version`` are available.

event=no-result
  No version is detected. There may be an error. Fields ``name`` is available.

level=error
  There is an error. Fields ``name`` and ``exc_info`` may be available to give
  further information.

Version Record Files
====================
Version record files record which version of the software you know or is available. They are simple key-value pairs of ``(name, version)`` separated by a space::

  fcitx 4.2.7
  google-chrome 27.0.1453.93-200836
  vim 7.3.1024

Say you've got a version record file called ``old_ver.txt`` which records all your watched software and their versions, as well as some configuration entries. To update it using ``nvchecker``::

  nvchecker source.ini

See what are updated with ``nvcmp``::

  nvcmp source.ini

Manually compare the two files for updates (assuming they are sorted alphabetically; files generated by ``nvchecker`` are already sorted)::

  comm -13 old_ver.txt new_ver.txt
  # or say that in English:
  comm -13 old_ver.txt new_ver.txt | awk '{print $1 " has updated to version " $2 "."}'
  # show both old and new versions
  join old_ver.txt new_ver.txt | awk '$2 != $3'

The ``nvtake`` Command
----------------------
This command helps to manage version record files. It reads both old and new version record files, and a list of names given on the commandline. It then update the versions of those names in the old version record file.

This helps when you have known (and processed) some of the updated software, but not all. You can tell nvchecker that via this command instead of editing the file by hand.

This command will help most if you specify where you version record files are in your config file. See below for how to use a config file.

Version Source Files
====================
The software version source files are in ini format. *Section names* is the name of the software. Following fields are used to tell nvchecker how to determine the current version of that software.

See ``sample_source.ini`` for an example.

Configuration Section
---------------------
A special section named ``__config__`` is special, it provides some configuration options.

Relative path are relative to the source files, and ``~`` and environmental variables are expanded.

Currently supported options are:

oldver
  Specify a version record file containing the old version info.

newver
  Specify a version record file to store the new version info.

proxy
  The HTTP proxy to use. The format is ``host:port``, e.g. ``localhost:8087``.

max_concurrent
  Max number of concurrent jobs. Default: 20.

keyfile
  Specify an ini config file containing key (token) information. This file should contain a ``keys`` section, mapping key names to key values. See specific source for the key name(s) to use.

Global Options
--------------
The following options apply to all checkers.

prefix
  Strip the prefix string if the version string starts with it. Otherwise the
  version string is returned as-is.

from_pattern, to_pattern
  Both are Python-compatible regular expressions. If ``from_pattern`` is found
  in the version string, it will be replaced with ``to_pattern``.

missing_ok
  Suppress warnings and errors if a version checking module finds nothing.
  Currently only ``regex`` supports it.

If both ``prefix`` and ``from_pattern``/``to_pattern`` are used,
``from_pattern``/``to_pattern`` are ignored. If you want to strip the prefix
and then do something special, just use ``from_pattern```/``to_pattern``. For
example, the transformation of ``v1_1_0`` => ``1.1.0`` can be achieved with
``from_pattern = v(\d+)_(\d+)_(\d+)`` and ``to_pattern = \1.\2.\3``.

Search in a Webpage
-------------------
Search through a specific webpage for the version string. This type of version finding has these fields:

url
  The URL of the webpage to fetch.

encoding
  (*Optional*) The character encoding of the webpage, if ``latin1`` is not appropriate.

regex
  A regular expression used to find the version string.

  It can have zero or one capture group. The capture group or the whole match is the version string.

  When multiple version strings are found, the maximum of those is chosen.

proxy
  The HTTP proxy to use. The format is ``host:port``, e.g. ``localhost:8087``.

user_agent
  The ``User-Agent`` header value to use. Use something more like a tool (e.g. ``curl/7.40.0``) in Europe or the real web page won't get through because cookie policies (SourceForge has this issue).

sort_version_key
  Sort the version string using this key function. Choose between ``parse_version`` and 
  ``vercmp``. Default value is ``parse_version``. ``parse_version`` use 
  ``pkg_resources.parse_version``. ``vercmp`` use ``pyalpm.vercmp``.

Find with a Command
-------------------
Use a shell command line to get the version. The output is striped first, so trailing newlines do not bother.

cmd
  The command line to use. This will run with the system's standard shell (i.e. ``/bin/sh``).

Check AUR
---------
Check `Arch User Repository <https://aur.archlinux.org/>`_ for updates.

aur
  The package name in AUR. If empty, use the name of software (the *section name*).

strip-release
  Strip the release part.

Check GitHub
------------
Check `GitHub <https://github.com/>`_ for updates. The version returned is in date format ``%Y%m%d``, e.g. ``20130701``.

github
  The github repository, with author, e.g. ``lilydjwg/nvchecker``.

branch
  Which branch to track? Default: ``master``.

use_latest_release
  Set this to ``true`` to check for the latest release on GitHub. An annotated
  tag creates a "release" on GitHub. It's not the same with git tags, which
  includes both annotated tags and lightweight ones.

use_max_tag
  Set this to ``true`` to check for the max tag on GitHub. Unlike ``use_latest_release``,
  this option includes both annotated tags and lightweight ones, and return the biggest one
  sorted by ``pkg_resources.parse_version``.

include_tags_pattern
  Only include tags matching this pattern. This option must be used together
  with ``use_max_tag``.

ignored_tags
  Ignore certain tags while computing the max tag. Tags are separate by
  whitespaces. This option must be used together with ``use_max_tag``. This can
  be useful to avoid some known badly versioned tags, so the newer tags won't
  be "overridden" by the old broken ones.

sort_version_key
  Sort the version string using this key function. Choose between ``parse_version`` and 
  ``vercmp``. Default value is ``parse_version``. ``parse_version`` use 
  ``pkg_resources.parse_version``. ``vercmp`` use ``pyalpm.vercmp``.

proxy
  The HTTP proxy to use. The format is ``host:port``, e.g. ``localhost:8087``.

An environment variable ``NVCHECKER_GITHUB_TOKEN`` or a key named ``github``
can be set to a GitHub OAuth token in order to request more frequently than
anonymously.

Check BitBucket
---------------
Check `BitBucket <https://bitbucket.org/>`_ for updates. The version returned
is in date format ``%Y%m%d``, e.g. ``20130701``.

bitbucket
  The bitbucket repository, with author, e.g. ``lilydjwg/dotvim``.

branch
  Which branch to track? Default is the repository's default.

use_max_tag
  Set this to ``true`` to check for the max tag on BitBucket. Will return the biggest one
  sorted by ``pkg_resources.parse_version``.

ignored_tags
  Ignore certain tags while computing the max tag. Tags are separate by
  whitespaces. This option must be used together with ``use_max_tag``. This can
  be useful to avoid some known badly versioned tags, so the newer tags won't
  be "overridden" by the old broken ones.

sort_version_key
  Sort the version string using this key function. Choose between ``parse_version`` and 
  ``vercmp``. Default value is ``parse_version``. ``parse_version`` use 
  ``pkg_resources.parse_version``. ``vercmp`` use ``pyalpm.vercmp``.

Check GitLab
-------------
Check `GitLab <https://gitlab.com/>`_ for updates. The version returned is in date format ``%Y%m%d``, e.g. ``20130701``.

gitlab
  The gitlab repository, with author, e.g. ``Deepin/deepin-music``.

branch
  Which branch to track? Default: ``master``.

use_max_tag
  Set this to ``true`` to check for the max tag on BitBucket. Will return the biggest one
  sorted by ``pkg_resources.parse_version``.

ignored_tags
  Ignore certain tags while computing the max tag. Tags are separate by
  whitespaces. This option must be used together with ``use_max_tag``. This can
  be useful to avoid some known badly versioned tags, so the newer tags won't
  be "overridden" by the old broken ones.

sort_version_key
  Sort the version string using this key function. Choose between ``parse_version`` and 
  ``vercmp``. Default value is ``parse_version``. ``parse_version`` use 
  ``pkg_resources.parse_version``. ``vercmp`` use ``pyalpm.vercmp``.

host
  Hostname for self-hosted GitLab instance.

token
  GitLab authorization token used to call the API.
  
Authenticated only.

To set a authorization token, you can set:

- a key named ``gitlab_{host}`` in the keyfile (where ``host`` is formed the
  same as the environment variable, but all lowercased).
- an environment variable ``NVCHECKER_GITLAB_TOKEN_{host}`` must provide that
  token. The ``host`` part is the uppercased version of the ``host`` setting,
  with dots (``.``) and slashes (``/``) replaced by underscores (``_``), e.g.
  ``NVCHECKER_GITLAB_TOKEN_GITLAB_COM``.
- the token option

Check PyPI
----------
Check `PyPI <https://pypi.python.org/>`_ for updates.

pypi
  The name used on PyPI, e.g. ``PySide``.

Check RubyGems
--------------
Check `RubyGems <https://rubygems.org/>`_ for updates.

gems
  The name used on RubyGems, e.g. ``sass``.

Check NPM Registry
------------------
Check `NPM Registry <https://registry.npmjs.org/>`_ for updates.

npm
  The name used on NPM Registry, e.g. ``coffee-script``.

Check Hackage
-------------
Check `Hackage <https://hackage.haskell.org/>`_ for updates.

hackage
  The name used on Hackage, e.g. ``pandoc``.

Check CPAN
--------------
Check `MetaCPAN <https://metacpan.org/>`_ for updates.

cpan
  The name used on CPAN, e.g. ``YAML``.

proxy
  The HTTP proxy to use. The format is ``host:port``, e.g. ``localhost:8087``.

Check Packagist
---------------
Check `Packagist <https://packagist.org/>`_ for updates.

packagist
  The name used on Packagist, e.g. ``monolog/monolog``.

Check Local Pacman Database
---------------------------
This is used when you run ``nvchecker`` on an Arch Linux system and the program always keeps up with a package in your configured repositories for `Pacman`_.

pacman
  The package name to reference to.

strip-release
  Strip the release part.

Check Arch Linux official packages
----------------------------------
This enables you to track the update of `Arch Linux official packages <https://www.archlinux.org/packages/>`_, without needing of pacman and an updated local Pacman databases.

archpkg
  Name of the Arch Linux package.

strip-release
  Strip the release part.

Check Debian Linux official packages
----------------------------------
This enables you to track the update of `Debian Linux official packages <https://packages.debian.org>`_, without needing of apt and an updated local APT database.

debianpkg
  Name of the Debian Linux source package.

suite
  Name of the Debian release (jessie, wheezy, etc, defaults to sid)

strip-release
  Strip the release part.

Check Ubuntu Linux official packages
------------------------------------
This enables you to track the update of `Ubuntu Linux official packages <https://packages.ubuntu.com/>`_, without needing of apt and an updated local APT database.

ubuntupkg
  Name of the Ubuntu Linux source package.

suite
  Name of the Ubuntu release (xenial, zesty, etc, defaults to None, which means no limit on suite)

strip-release
  Strip the release part.

Check Anitya
------------
This enables you to track updates from `Anitya <https://release-monitoring.org/>`_ (release-monitoring.org).

anitya
  ``distro/package``, where ``distro`` can be a lot of things like "fedora", "arch linux", "gentoo", etc. ``package`` is the package name of the chosen distribution.

Check Android SDK
-----------------
This enables you to track updates of Android SDK packages listed in ``sdkmanager --list``.

android_sdk
  The package path prefix. This value is matched against the ``path`` attribute in all <remotePackage> nodes in an SDK manifest XML. The first match is used for version comparisons.

repo
  Should be one of ``addon`` or ``package``. Packages in ``addon2-1.xml`` use ``addon`` and packages in ``repository2-1.xml`` use ``package``.

Manually updating
-----------------
This enables you to manually specify the version (maybe because you want to approve each release before it gets to the script).

manual
  The version string.

Version Control System (VCS) (git, hg, svn, bzr)
------------------------------------------------
Check a VCS repo for new commits. The version returned is currently not related to the version of the software and will increase whenever the referred VCS branch changes. This is mainly for Arch Linux.

vcs
  The url of the remote VCS repo, using the same syntax with a VCS url in PKGBUILD (`Pacman`_'s build script). The first VCS url found in the source array of the PKGBUILD will be used if this is left blank. (Note: for a blank ``vcs`` setting to work correctly, the PKGBUILD has to be in a directory with the name of the software under the path where nvchecker is run. Also, all the commands, if any, needed when sourcing the PKGBUILD need to be installed).

use_max_tag
  Set this to ``true`` to check for the max tag. Currently only supported for ``git``.
  This option returns the biggest tag sorted by ``pkg_resources.parse_version``.

ignored_tags
  Ignore certain tags while computing the max tag. Tags are separate by
  whitespaces. This option must be used together with ``use_max_tag``. This can
  be useful to avoid some known badly versioned tags, so the newer tags won't
  be "overridden" by the old broken ones.

Other
-----
More to come. Send me a patch or pull request if you can't wait and have written one yourself :-)

Bugs
====
* Finish writing results even on Ctrl-C or other interruption.

Footnotes
=========

.. _Pacman: https://wiki.archlinux.org/index.php/Pacman