typemap in perl/xs
DESCRIPTION
for OSDC.TW 2013TRANSCRIPT
Wine label from Perl (located in Germany)
Thanks , Wendy and Liz!
... and MST
typemap in Perl/XS
Kenichi Ishigaki(charsbar)
@OSDC.TW 2013April 19, 2013
Let me ask you first.
Have you ever written an
extension that bridges C and Perl?
Part I: XS is ...?
Perl has a feature to load an external C
library.
use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
This won't work correctly unless the library conforms
with Perl's convention.
use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
So we need something in-
between.
Perl XS C library
Writing the one-in-between is fairly
easy.
(as long as you don't add
anything extra)
The simplest form of the one-in-between starts like
this:#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>
MODULE = Foo PACKAGE = Foo
intfunc(const char* str)
These three lines are to use Perl API.
#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>
MODULE = Foo PACKAGE = Foo
intfunc(const char* str)
This is for portability between perls.
#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>
MODULE = Foo PACKAGE = Foo
intfunc(const char* str)
This is to import C functions from the library.
#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>
MODULE = Foo PACKAGE = Foo
intfunc(const char* str)
Module and package declarations.
#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>
MODULE = Foo PACKAGE = Foo
intfunc(const char* str)
The function declarations to export (XSUBs) follow.
#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>
MODULE = Foo PACKAGE = Foo
intfunc(const char* str)
As with .h files, there should be only declarations.
#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>
MODULE = Foo PACKAGE = Foo
intfunc(const char* str)
Nothing difficult (in principle).
The library you use may not always be
that simple.
People may add something written in C with a variety of
Perl API.
If someone's XS code scares you, please remember.
What's difficult is not the XS
interface itself.
It's something else that makes
the matter complicated.
Don't hesitate writing XS interface if you find a useful
C library.
Part II: How .xs file is compiled
To build an interface library, we need to turn its XS
declaration into pure C code.
If you already have Makefile.PL, run it, and
then, make.
$ perl Makefile.PL && make
If you have Build.PL, do something like this (or maybe ./Build build).
$ perl Build.PL && ./Build
If you don't have either, Milla or Minilla will help
you (ask miyagawa-san).
The most important XSUB part will be translated like
this:XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}
A function exposed to the Perl world takes a Perl variable as its
argument.XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}
However, a Perl variable is actually a structure in C.
XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}
You can't pass it directly to a C function, and vice versa.
XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}
The Perl variable is converted and cast into a C value here.
XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}
Then, the function imported from the library is called.
XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}
And the return value is converted into a Perl value
again.XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}
typemap is the key to this conversion.
Basic type mappings are
defined in ExtUtils::typemap
.$ perldoc -m ExtUtils::typemap
# basic C typesint T_IVunsigned T_UVunsigned int T_UVlong T_IVunsigned long T_UVshort T_IVunsigned short T_UVchar T_CHARunsigned char T_U_CHARchar * T_PVunsigned char * T_PVconst char * T_PVcaddr_t T_PVwchar_t * T_PVwchar_t T_IV
# bool_t is defined in <rpc/rpc.h>bool_t T_IVsize_t T_UVssize_t T_IVtime_t T_NVunsigned long *T_OPAQUEPTRchar **T_PACKEDARRAYvoid * T_PTRTime_t * T_PVSV * T_SV
Any type not listed there should be
added locally, in a file named "typemap".
Part III: On type mapping
A typical "typemap" file
has three sections.
"TYPEMAP" section contains a pair of C type
and XS type.
TYPEMAPchar * T_PV
How to map should be described in the "INPUT" and
"OUTPUT" sections, if necessary.
INPUTT_PV $var = ($type)SvPV_nolen($arg)
OUTPUTT_PV sv_setpv((SV*)$arg, $var);
These code fragments will be interpolated while processing an XS file.
INPUTT_PV $var = ($type)SvPV_nolen($arg)
OUTPUTT_PV sv_setpv((SV*)$arg, $var);
Adding orphan C types is easy.
TYPEMAPmy_id T_IVmy_str T_PV
Modifying the behavior of an existing type needs some
consideration.
If you are confident, just add a new code
fragment in your local typemap.
INPUTT_PV if (!SvOK($arg)) { $var = NULL; } else { $var = ($type)SvPV_nolen($arg); }
Or, you might want to add another typedef in the XS file, before MODULE and PACKAGE
declarations.
typedef char * my_nullable_str;
MODULE Foo PACKAGE FOO
intfunc(my_nullable_str str)
And then, add a behavior of this new type in your local
typemap.TYPEMAPmy_nullable_str T_PV_OR_NULL
INPUTT_PV_OR_NULL if (!SvOK($arg)) { $var = NULL; } else { $var = ($type)SvPV_nolen($arg); }
There is also a much trickier way to do it.
MODULE Foo PACKAGE FOO
intfunc(char * str_or_null)
If you don't get why this works, see
perlref.INPUTT_PV $var = ${ $var =~ /_or_null$/ ? \qq{NULL} : \qq{($type)SvPV_nolen($arg)} }
Remove a star and prepend "OUT" (or
"IN_OUT") for an argument called by reference.
MODULE Foo PACKAGE FOO
intfunc(char * str, OUT int len)
Further Reading
• perlxs• perlxstut• perlxstypemap• perlapi• perlcall• ... and others' XS
code :p
Part IV: From .h files to XS files
We've written XS files by hand so
far.
Of course this can be done with a
tool.
The best-known tool is h2xs.
It's mainly used to expose
constants from a C library.
With the help of C::Scan, you can use
it to expose C functions as well.
$ h2xs -Axan Foo /path/to/header.h
Defaulting to backwards compatibility with perl 5.xx.xIf you intend this module to be compatible with earlier perl versions, please specify a minimum perl version with the -b option.
Writing Foo/ppport.hScanning typemaps... Scanning /home/xxxx/perl5/perlbrew/perls/perl-5.xx.x/lib/5.xx.x/ExtUtils/typemapScanning /path/to/header.h for functions...Scanning /path/to/header.h for typedefs...Writing Foo/lib/Foo.pmWriting Foo/Foo.xsWriting Foo/typemapWriting Foo/Makefile.PLWriting Foo/READMEWriting Foo/t/Foo.tWriting Foo/ChangesWriting Foo/MANIFEST
If the library is stable, this may give you a good starting
point.
However.
The files h2xs generates are rather
old-fashioned.
It's not good at updating only a part
of a distribution, either.
I started writing Convert::H::XS.
https://github.com/charsbar/convert_h_xs
Convert::H::XS takes a C header file, and looks for the minimum
information to write XS components.
use Convert::H::XS;my $converter = Convert::H::XS->new;$converter->process($h_file);
Convert::H::XS also provides methods to write XS
components for convenience.
$converter->write_constants( "xs/constants.inc",);
You can tweak those pieces of information
with a callback.
$converter->write_functions( "xs/functions.inc", sub { my ($type, $name, $args) = @_; ... return ($type, $name, $args);});
Grateful if you help me improve
this at the hackathon.
Part V: CPANTS update
I talked about CPANTS last year.
CPANTS tests what module authors often
forget to test by themselves.
Perl QA Hackathon 2013
in Lancaster
Several metrics are going to be removed.
Several metrics are going to be added.
To add new metrics, we need to
find issues.
Issues are often raised by other QA/toolchain
people.
• Do not ship modules with Module::Install 1.04http://weblog.bulknews.net/post/33907905561/do-not-ship-modules-with-module-install-1-04
• stop shipping MYMETA to CPANhttp://weblog.bulknews.net/post/44251476706/stop-shipping-mymeta-to-cpan
We need to confirm they are
measurable, and widespread.
What can I use to confirm?
• CPANTS databases
• metacpan• CPAN grep
I just wanted something else.
Groonga
• a fulltext search engine• a column store• actively developed (monthly
release)
• client/server (http/gqpt)• C API• Web interface
Groonga::API
https://github.com/charsbar/groonga-api
Also grateful if you help me improve
this at the hackathon.
Thank you!