The access.conf has all information about users, profiles, authentication and accounting. For general system configuration use the NntpswitchConf. This file is a brief explanation of the access.conf. Each option and parameter should be defined before used.
Options can be listed only in this order:See also example access.conf
wildmat all * wildmat none !* wildmat nl nl.* wildmat big8 sci.*,comp.*,news.*,talk.*,misc.*,soc.*,rec.*,humanities.* wildmat local news-service.*
args defaultdb host=localhost user=example password=example
# Always create a reasonable default alias auth_default auth_passwd.so /usr/local/etc/nntpswitch/passwd alias acct_default acct_syslog.so local6 # Plaintext password file #alias authCustomer1 auth_passwd.so /usr/local/etc/nntpswitch/passwd.cust1 # POP3 relay #alias authSupportNet auth_pop3.so mail.support.net:110 # NNTP relay authentication #alias authNewsService auth_remote.so news.news-service.com:119 # Radius #alias authRadius1 auth_radius.so radius.example.com:secretexample #alias acctRadius1 acct_radius.so radius.example.com:secretexample # Postgres #alias authPostgres auth_postgres.so dbname=users #alias authPgSQL2 auth_postgres.so hostaddr=127.0.0.1 port=5432 dbname=users user=test password=test # MySQL #alias authMySQL1 auth_mysql.so hostname=localhost,dbname=users #alias authMySQL2 auth_mysql.so hostname=localhost dbname=test username=test password=test # LDAP #alias authLDAP auth_ldap.so # no options, hack auth_ldap.c # A parameter of "true" always returns Authentication OK # Any other parameter always returns authentication rejected #alias authTrue auth_ignore.so true #alias authFalse auth_ignore.so # Some Accounting aliases #alias acctAuthInfo acct_syslog.so auth.info #alias acctLocal4 acct_syslog.so local4 #alias acctNone acct_none.so
#
# Profiles. Always leave the "default" profile!
# If a user connects, a profile is assigned based on the ip address (see
# access.conf) If after that a user authenticates (see auth.conf) a
# different profile can be assigned.
#
profile default
# Patterns are defined using "wildmats" above
ReadPat none
PostPat none
# Disable this profile. Used in some authentication modules.
Disabled 0
# Allow SSL on this profile. Only supported in the commercial version.
# AllowSSL 0
# Which type of accounting to use in this profile. This field is required.
Accounting acct_default
# Put replacebanner to 1 to only see this banner and
# not the connection and version stuff.
# This only works for acl based connections.
Banner www.News-Service.com
ReplaceBanner 0
# Footer to add to every post. Leave blank to have no footer
# FooterFile /etc/nntpswitch/spam.txt
# Global max
MaxConnections 500
MaxUsers 50
# Allow users to connect only from 1 ip address.
SingleHostUser 1
# Max per user
MaxHostConcurrent 4
MaxUserConcurrent 4
MaxSessionTime 86400
MaxSessionBytes 1000000000
# Limits. Enforce fair use policy by slowing connection down after X gb.
# Supported in commercial version of nntpswitch and needs an external
# accounting database with some scripts.
Limits 0 0
# Kill connection after this many seconds
ClientReadTimeout 180
# Those 3 are practically the same. Pick one for the user speed
# MaxUserBPS 125000
# MaxUserKbit 1000
MaxUserMbit 1
# Same here, pick one but for all users together in a this profile
# MaxProfileBPS 3125000
# MaxProfileKbit 25000
MaxProfileMbit 25
# Stuff to put in headers of posts
XComplaintsTo abuse@example.com
Hostname news.example.com
Organization www.News-Service.com
# Force to 1 to overwrite user specified Organization: header
ForceOrganization 1
# Add nntp-posting-host and nntp-posting-date to posts
AddNNTPPostingHost 1
# Whether we allow control messages or not
NoControl 0
# Acct as transparant ihave slave.
Slave 0
# Allow article lookups by message-id without joining a group first
# This can force more connects and history lookups on your spoolservers.
ArticleSearch 1
# Ratelimit tuning. This is the factor to increase or decrease the delay
# each time the current bandwidth drops below or above the specified
# MaxLimits values. The default config says reach the maximum bandwidth
# faster then slowing it down. Normally not needed tuning unless you
# need kbit resolution on a gbit scale.
UserRLFactorU 1.6
UserRLFactorD 1.4
ProfileRLFactorD 1.6
ProfileRLFactorU 1.4
end
# all other profiles are inherited from profile "default"
# only list options different from default profile
profile world
ReadPat local
PostPat none
MaxSessionTime 7200
MaxConnections 500
end
profile big8
ReadPat big8
PostPat none
MaxSessionTime 86400
MaxConnections 500
end
# # Includes can only be at this point, after the default profile has been specified # and before the default authentication and acl mapping. # # include customers/news-service.conf # include customers/customer1.conf
# # Authentication Configuration # # Use a pattern to match authorization aliases # A "pattern" must have one *, actually its a wildmat pattern # Read/Post patterns are wildmats names specified in access.conf # Specifying them here will overrule the one from access.conf # If you specify no profile, it defaults to the default profile. # # auth mask auth-alias profile #auth *@support.net authSupportNet #auth *@news-service.com authNewsService #auth cust1_* authCustomer1 default # Always leave 1 "auth *" line, alias names are auth_default and acct_default, by default # This is a catch-all line if no patterns are specified auth * auth_default world # Or this will do the same, using all defaults #auth *
# # ACL Configuration # # # Possible options are: # read - allow read access # post - allow post access # auth - force authentication # stats - possibility to use statistics command # unlimit - don't enforce concurrent connections limits # apost - allow approved postings. if you don't specifiy this nntpswitch # will strip the Approved: header and so newgroup and rmgroup # messages will not work. # # The format file is: # CIDR/range Options Profile # # If no profile is specified it will use "default" profile. # acl 127.0.0.0/16 read,post,apost,stats,unlimit acl 192.168.0.0/16 read,post # Always use a 0/0 entry as the default! acl 0/0 deny # Everyone has read and post rights after authentication #acl 0/0 read,post,auth world # Let everyone just read big8 without posting rights #acl 0/0 read big8
There are a few possible errors endusers can get upon connecting with a NNTPSwitch server.
This is most likely a wrong username and/or password.
This means that the authentication server can not be reached. Check your error logs (<tt>news.err</tt>) for details. This is either a misconfigured authentication option, or a temporarily network or system error.
This means the profile the user should be put into is full. Probably the MaxProfileConnections is too low.
Same as above, but the number of users for the profile is full instead of the number of connections. Check MaxProfileUsers.
The enduser is opening too many connections. Each user can normally open 10 or 20 concurrent connections. Check MaxUserConccurent
The enduser is opening too many connections or multiple users are connected from the same host. Each user can normally open 10 or 20 concurrent connections. Check MaxHostConccurent in this case.
This happens if the SingleHostUser option is set and the user is already logged in from another host.
If a user connects using acl based access most of the same errors can appear, but with a 400 error code.
There is no current stable release. The latest stable version is always the cvs version. There is no big development going on at this moment however, small bugs and fixes are applied on a regular base. You can Browse the cvs repository to see the newest changes.
export CVSROOT=:pserver:cvs@cvs.chiparus.net:/usr/local/cvs cvs login use any password cvs co nntpswitch
A Linux system is required! Portability is not a priority at the moment. To estimate system requirements you need to know the maximum number of concurrent users you will have. A p3/1.3ghz can handle about 1500 concurrent broadband connections doing about 250 mbps. If you have dialup users you'll probably can handle 3000 concurrent connections. A modern 3ghz dual core system should be able to saturate a 1 gigabit line easy with 3-4000 connections.
Memory footprint is average to high. NNTPSwitch needs about 500k for each client and a shared space of about 30mb for roughly 65000 groups (60mb for 100k groups).
If you are running a local dreader to handle spoolservers performance is less of an issue because dreader will take up most resources.
You should have Perl installed in your $PATH.
make
make installmake calls make config automatically. make config parses the nntpswitch.conf.in file and uses these values as defaults.
Currently the makefile needs the mysql-client, pgsql and libradius installed. If you don't want this you can comment this out in the Makefile.
If you run debian you can install all prerequisites except radius by issuing:
apt-get -u install libmcrypt-dev libldap2-dev libmysqlclient15-dev postgresql-devCreate a directory /etc/nntpswitch or /usr/local/etc/nntpswitch (or whatever else is specified in config.c) and copy nntpswitch.conf, etc/access.conf etc/groups.conf etc/auth.conf and etc/overview.fmt to your /etc/nntpswitch directory.
mkdir /etc/nntpswitch
cp nntpswitch.conf-dist /etc/nntpswitch/nntpswitch.conf
cp etc/overview.fmt /etc/nntpswitch/
cp etc/servers.conf /etc/nntpswitch/
cp etc/access.conf /etc/nntpswitch/Edit nntpswitch.conf, servers.conf and access.conf to match your configuration.
If you're finished run updategroups for the first time to create the active, newsgroups and active.times from the backend servers.
/usr/local/sbin/updategroupsTo test all configuration you can run nntpswitch in test mode:
/usr/local/sbin/nntpswitchd -tIf everything looks ok, you can start it up with no parameters, or use the init.d script.
You need to have Time::ParseDate installed in order for the Perl statistics script to work. See Statistics for details.
The radius authorization and accounting modules depend on FreeBSD/Juniper libradius. The freeradius libradius does not work currently.
Download Download Libradius for Linux
You need to compile libmd first, then libradius. If you're running on a 64bit system add -fPIC to the CFLAGS variable in the makefile of both packages.
The postgres module depends on libpq-fe. Mysql module depends on libmysqlclient. Both should be installed by your database development package.
You should run statistics, rotate your logfiles, and update newsgroups lists at least once a day. It's recommended you run the nsstats.sh shell script from cron every 24hr. The nsstats.sh does the following things:
This was formerly the BUGS file. It's a list of things currently broken, wanted features and misc thoughts.
Stats from 31-12-xxxx created on 01-01-yyyy will carry the year yyyy instead of xxxx.
Cleanfeed is now done by connecting to an external INN server for example.
Don't know if i should implement full perl handling or create a wrapper
shell script around Cleanfeed. If we start to spool posts we could as well
handle some incoming feed. A headerfeed would be nice to remember
message-id and group combinations.
IPV6 and possibly other methods of connecting and authenticating should be
build. Probably configurable so when for example ipv6 is configured all the
configuration should be build around an ipv6 module for example.
Serverhandling in general could be more transparant so the post command
doesn't have to reconnect every post for example. A dream is to open
multiple server slots and have different clients connect to different
server slots. Server slots could be reused.
We currently do not know if the clients needs feedback if a remote server
is down. If we're in the middle of an article its better to just drop
the connection instead of appending some errormessage. Also by the time
a server times out, most clients already have timed out theirselves. These
entries are marked FIXME in socket.c
Timezones have to be implented for commands using timestrings. Currently
it default to UTC hardcoded most of the time. Commands affected: newnews,
newgroups (no TZ specified by user is current time, GMT specified will
use GMT/UTC time).
Should be done sometime! See general server handling. Some really stupid clients
using HELP command to prevent idle timeout. This will kill the backend server
connection. There should be 'current server' flag somewhere and check for
each command if it needs changing (for example checking 'current group',
which should be id'ified in itself). Xover and Post command should use
server based on group list or balance policy. Yes, post server balancing and
backup is required. Servers.conf reading has already support for post and xover
type servers, however it's not used in other places of the code yet.
Newsgroups descriptions with the word "description" in their description
are vanished. Time for regexp libs? (authentication could benefit from that)
When transferring users from one profile to another, i.e. when
editing the profiles.conf and reloading the master server. The users
are not disconnected from their current profile, instead, they're
tranferred to the new profile, without fixing the counters. The new
profile could then have a negative amount of users, resulting in
'too many users in your class'.
This happens when removing or adding ACL entries probably. Still
its a bit unclear as to how this happens here. Should be looked into.
Its nice, but leaks mem on every config (re)load (replace_path).
Create config options for timeouts and/or rotate resolver options.
Make this more configurable.
Authorization and accounting should be made redundant. Probably by allowing multiple auth entries with each a different authentication modules, or options. Accounting is only redundant in commercial version with the radtool and retry scripts.
This Changelog list changes from the old 0.11 version to the new current 0.12 version.
Your old config is likely to be incompatible due to the new accounting configuration. Mainly the accounting configuration moved from the acl/auth entries to the profile.
To update, first add the Accounting keyword to your default profile, pointing to your default accounting module. Next, remove them from the auth and acl config.
If you have something like this:auth * authname acctname profile acl 0/0 acctname profileChange it to:
auth * authname profile acl 0/0 profileAnd put this in your profiles:
Accounting acctname
The levels have been removed from the servers.conf. To upgrade, just delete the level keywords from your servers.conf.
NNTPSwitch is a High Performance Linux NNTP Content Router. It can act as a loadbalancer and virtual hosting application. It's aimed at high-performance news server clusters for ISPs and Usenet resellers.
Depending on the backend server types, all NNTP commands and extensions are supported. NNTPSwitch performs access, authorization and accounting processes for any serious usenet reseller. It can handle anything from transparant home proxies to full featured multi gigabit usenet clusters.
NNTPSwitch currently supports PostgreSQL, MySQL, Radius, OpenLDAP, NNTP, POP3 and Plaintext authentication options. Accounting is supported using Radius or Syslog.
Documentation is always a work in progress. If you have any questions don't hesitate to contact me.
This file holds the main configuration options, paths, ports, ip addresses, logging and system tuning.
Default values from nntpswitch.conf are displayed separately like this.
BindAddress: The address of an interface to bound to, or 0.0.0.0 for all interfaces.ListenPorts 25 80 119 1199
RunAsUser: The user to run as. This user should be listed in /etc/passwd and it's homedir should hold all configuration for nntpswitch. Most of the time this will be user news. If you're running in conjunction with other news software you could create a separate user.BindAddress 0.0.0.0
PidFile: The place to store the pid file. If you're using the StatisticsTimer then only the pid from the main nntpswitch process in written in the pidfile. Also the pidfile is created as user root on startup and change ownership to RunAsUser after dropping privileges.RunAsUser news
PidFile /var/run/nntpswitchd.pid
AccessFile: The filename of the access configuration (mostly access.conf). If the filename starts with a / it is considered a full pathname, otherwise BasePath is prepended.BasePath /etc/nntpswitch
ServerFile: The server configuration file.AccessFile access.conf
OverviewFmtFile: This file specifies the overview format. Mostly not used, and most newsreaders ignore this anyway.ServerFile servers.conf
ActiveFile: The name of the active file. This file is generated by the updategroups command.OverviewFmtFile overview.fmt
NewsgroupsFile: The name of the newsgroups file. This file holds all newsgroup descriptions and is also maintained by the updategroups command.ActiveFile active
StatsFilePeriod: How many times to write the statistics file. This file can be used for rrdtool for example. Value in seconds.NewsgroupsFile newsgroups
StatsFile: The absolute location of the statsfile. This should have the full path name.StatsFilePeriod 60
StatsFile /tmp/.nntpswitch.stats
ServerConnectTimeout: Timeout for server connections.ServerReadTimeout 300
ClientReadTimeout: Timeout for client considered dead.ServerConnectTimeout 4
ClientReadTimeout 300
LogReadserver: Log everything read from the server, including articles! For debugging only!LogWriteclient 0
LogWriteserver: Log everything written to the server.LogReadserver 0
LogCommands: Log detailed command statistics.LogWriteserver 0
LogMissing: Log missing articles.LogCommands 0
LogAuthInfo: Log authinfo commands.LogMissing 0
LogAuthInfo 1
PostServerPort: If the post server runs on another port, configure thisPostServer posthost.example.com
ModeratorDomain: NNTPSwitch doesnt support a moderators configuration. Instead it forwards everything to groupname@moderators.isc.orgPostServerPort 119
MailerPath: What too to use to send mail. Even if you use postfix or exim this command will probably be aliased by your system Installation.ModeratorDomain moderators.isc.org
MailerPath /usr/sbin/sendmail -oi -t
BufSize: Internal buffer size for misc stuff.MaxConnections 500
SockBufSize: Overwrite the size of the socket buffer. If you leave this at 0 the kernel will autoconfigure it.BufSize 8192
ListenBackLog: Number of connections to hold in the backlog before dropping them.SockBufSize 0
DownDelay: After a server down after this many seconds, how many seconds to backoff the client.ListenBackLog 128
RetryDelay: This is the delay for backing off clients after they reach their connection limit. This will force clients to correctly configure their limits.DownDelay 4
RatePeriod: Recalibrate the rate limiter 5 times a sec. Anything else will make it slightly inaccurate.RetryDelay 2
RatePeriod 5
DNSLookups: Wheater to enable reverse DNS lookups or not.DenyNoIndexField 1
DropUnknown: Drop unknown clients after X invalid commands. Mostly mail and webservers and such.DNSLookups 1
DisableFile: Touch this file to disable new connection to the server.DropUnknown 10
DisableText: And the text to display to the client when the DisableFile is there.DisableFile closed
NoSuchArticleRetries: Sometimes diablo reports 'no such article' while the article is there. This option specifies how many retries we should do before returning no such article to the client.DisableText 500 Sorry, temporarily network issues, try again later.
CoreBind: Bind each nntpswitch process to a separate cpu core. For each new connection a cpu core is choosen from the pool. Normally you don't need this option.NoSuchArticleRetries 5
CoreBind 0
LocalDreader 0
By default the configuration option DenyNoIndexField is true which means that XPAT and XHDR searches for headers which are not in the overview.fmt will be disabled. To enable searches for all possible headers fields put this option to false (0).
Allowing searches for arbitary headers mostly means the backend server has to traverse all articles in the spool looking for headers. Known headers can be looked up in the overview database instead of the spool. Traversing the spool can mean a drastic load on the backend server. This is true for INN, but Diablo indexes full headers on their overview servers so you can put this to false.
In short, NEWNEWS only works for single groups instead of wildmats. This is because of the same reason NNTPSwitch has to be connected to a backend server. Using wildmats could mean multiple server connections, but if you allow bin for a soiikserver and the NEWNEWS command queries only binary groups, it should work.
Those are implemented, but not tested.
All backend servers are configured in the servers.conf. As with most other configuration files there is one special server which is called default. The default server contains no Hostname or Groups statements, but contains all other possibly needed parameters.
The following example demonstrates the most basic configuration. A binaries server and a text server. The textserver also has the descriptions and knows about active.times.
server default Port 119 Timeout 300 Descriptions False ActiveTimes False Type Spool Policy Single SplitList False end server bunnies Groups *bin* Hostname bin.example.com end server text Groups * Hostname text.example.com end server active Groups *.* Hostname text.example.com Descriptions True ActiveTimes True end
Please make sure that you retrieve active.times and newsgroups information from atleast one server. This must be done by setting the options activetimes and descriptions to true. Also note that the most significant servers are listed at the bottom. If you have servers for single newsgroups or patterns thereof put them at the top.
SplitList If you have a backend server which doesn't support the LIST ACTIVE command with multiple patterns, set this option to True so multiple LIST commands will be done for each part of the pattern. For Diablo put it to 1, for INN put it to 0.
You can configure backup or balanced servers by using the same Groups values. Policy can be either backup or balance.
The backup and balancing checks take place upon connecting to a group. Only when a group is selected nntpswitch checks if we need to (re)connect to a different backend server. A backup server is used when the primary server fails, failure is tested on every connect and the primary server will be retried first at all times. The balancing takes place by just selecting a different server on each connect. Only if one of the balancing servers failes, nntpswitch just uses the backup mechanism to find another server.
An example of 2 backup servers. Each connection will try "movies1" first. In case of some connection or NNTP error, the next server "movies2" is tried.
server bin1 Hostname binaries1.local Groups * Policy backup end server bin2 Hostname binaries2.local Groups * Policy backup end
An example of 2 load-balanced servers. Each connection to the backend server will connect to the other one.
server bin1 Hostname binaries1.local Groups *bin* Policy balance end server bin2 Hostname binaries2.local Groups *bin* Policy balance end
Note in the above examples, the server "movies1" is most likely used for updating the groups with the updategroups command.
If you have a diablo based setup with hash based spools and standalone overview servers, you can use a local dreader to handle server connections. When someone issues a group command and retrieves an article, nntpswitch will connect to the spool server which matches the pattern. However if someone requests an article by Message-ID nntpswitch loops to all spoolservers to find the message-id. If you have a local dreader running on a different port you can use that dreader to handle message-id lookups. See also the LocalDreader nntpswitch.conf option
To setup a local dreader handling message-id's but no groups configure it like this:
server localhost
hostname 127.0.0.1
groups !*
port 120
policy single
end
NNTPSwitch can create two kind of statistics. A detailed usage report of the server is created every day from the maintenance scripts. Also realtime profile usage statistics are available using rrd.
Daily usage reports are usually generated from the nsstats.sh shell script every night. However you might create intermediate reports using a command like this:
nsstats.pl -d /html/www/dir < news.noticeHigh volume sites can generate a lot of logging overhead. You can tune the following options in nntpswitch.conf: LogCommands, LogWriteclient, LogReadserver and LogWriteserver.
RRD statistics by using RRDBrowse is unsupported as of January 2008 but the documentation is still listed here for reference.
Your RRD host should have stats access in an ACL, for example:
192.168.1.1/32 read,post,statsFor each profile you should create 3 nfo files. One for the number of active/connected users and connections, one for the bandwidth and one for the number of articles per second.
Example of Users nfo:
Type: ntsprofile Target: 119@your.server Profile: default Description: Default Profile Users
Example of Bandwidth nfo:
Type: ntsprofbw Target: 119@your.server Profile: default Description: Default Profile Bandwidth
Example of Articles nfo:
Type: ntsprofarts Target: 119@your.server Profile: default Description: Default Profile Articles per Second
Contact Tommy van Leeuwen, main developer of NNTPSwitch at tommy at nntpswitch dot org.
Commercial support contracts are available through News-Service.com Contact sales at news-service dot com for more information. Additionally News-Service.com can deliver pre-installed servers or complete high-performance news server clusters.