typemap in perl/xs

Wine label from Perl (located in Germany)

Thanks , Wendy and Liz!

... and MST

typemap in Perl/XS

Kenichi Ishigaki(charsbar)

@OSDC.TW 2013April 19, 2013

Let me ask you first.

Have you ever written an

extension that bridges C and Perl?

Part I: XS is ...?

Perl has a feature to load an external C

library.

use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);

This won't work correctly unless the library conforms

with Perl's convention.

use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);

So we need something in-

between.

Perl XS C library

Writing the one-in-between is fairly

easy.

(as long as you don't add

anything extra)

The simplest form of the one-in-between starts like

this:#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>

MODULE = Foo PACKAGE = Foo

intfunc(const char* str)

These three lines are to use Perl API.

#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>

MODULE = Foo PACKAGE = Foo

intfunc(const char* str)

This is for portability between perls.

#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>

MODULE = Foo PACKAGE = Foo

intfunc(const char* str)

This is to import C functions from the library.

#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>

MODULE = Foo PACKAGE = Foo

intfunc(const char* str)

Module and package declarations.

#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>

MODULE = Foo PACKAGE = Foo

intfunc(const char* str)

The function declarations to export (XSUBs) follow.

#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>

MODULE = Foo PACKAGE = Foo

intfunc(const char* str)

As with .h files, there should be only declarations.

#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>

MODULE = Foo PACKAGE = Foo

intfunc(const char* str)

Nothing difficult (in principle).

The library you use may not always be

that simple.

People may add something written in C with a variety of

Perl API.

If someone's XS code scares you, please remember.

What's difficult is not the XS

interface itself.

It's something else that makes

the matter complicated.

Don't hesitate writing XS interface if you find a useful

C library.

Part II: How .xs file is compiled

To build an interface library, we need to turn its XS

declaration into pure C code.

If you already have Makefile.PL, run it, and

then, make.

$ perl Makefile.PL && make

If you have Build.PL, do something like this (or maybe ./Build build).

$ perl Build.PL && ./Build

If you don't have either, Milla or Minilla will help

you (ask miyagawa-san).

The most important XSUB part will be translated like

this:XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}

A function exposed to the Perl world takes a Perl variable as its

argument.XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}

However, a Perl variable is actually a structure in C.

XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}

You can't pass it directly to a C function, and vice versa.

XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}

The Perl variable is converted and cast into a C value here.

XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}

Then, the function imported from the library is called.

XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}

And the return value is converted into a Perl value

again.XS_EUPXS(XS_Foo_func){ dVAR; dXSARGS; if (items != 1) croak_xs_usage(cv, "str"); { int RETVAL; dXSTARG; const char* str = (const char *)SvPV_nolen(ST(0)); RETVAL = func(str); XSprePUSH; PUSHi((IV)RETVAL); } XSRETURN(1);}

typemap is the key to this conversion.

Basic type mappings are

defined in ExtUtils::typemap

.$ perldoc -m ExtUtils::typemap

# basic C typesint T_IVunsigned T_UVunsigned int T_UVlong T_IVunsigned long T_UVshort T_IVunsigned short T_UVchar T_CHARunsigned char T_U_CHARchar * T_PVunsigned char * T_PVconst char * T_PVcaddr_t T_PVwchar_t * T_PVwchar_t T_IV

# bool_t is defined in <rpc/rpc.h>bool_t T_IVsize_t T_UVssize_t T_IVtime_t T_NVunsigned long *T_OPAQUEPTRchar **T_PACKEDARRAYvoid * T_PTRTime_t * T_PVSV * T_SV

Any type not listed there should be

added locally, in a file named "typemap".

Part III: On type mapping

A typical "typemap" file

has three sections.

"TYPEMAP" section contains a pair of C type

and XS type.

TYPEMAPchar * T_PV

How to map should be described in the "INPUT" and

"OUTPUT" sections, if necessary.

INPUTT_PV $var = ($type)SvPV_nolen($arg)

OUTPUTT_PV sv_setpv((SV*)$arg, $var);

These code fragments will be interpolated while processing an XS file.

INPUTT_PV $var = ($type)SvPV_nolen($arg)

OUTPUTT_PV sv_setpv((SV*)$arg, $var);

Adding orphan C types is easy.

TYPEMAPmy_id T_IVmy_str T_PV

Modifying the behavior of an existing type needs some

consideration.

If you are confident, just add a new code

fragment in your local typemap.

INPUTT_PV if (!SvOK($arg)) { $var = NULL; } else { $var = ($type)SvPV_nolen($arg); }

Or, you might want to add another typedef in the XS file, before MODULE and PACKAGE

declarations.

typedef char * my_nullable_str;

MODULE Foo PACKAGE FOO

intfunc(my_nullable_str str)

And then, add a behavior of this new type in your local

typemap.TYPEMAPmy_nullable_str T_PV_OR_NULL

INPUTT_PV_OR_NULL if (!SvOK($arg)) { $var = NULL; } else { $var = ($type)SvPV_nolen($arg); }

There is also a much trickier way to do it.

MODULE Foo PACKAGE FOO

intfunc(char * str_or_null)

If you don't get why this works, see

perlref.INPUTT_PV $var = ${ $var =~ /_or_null$/ ? \qq{NULL} : \qq{($type)SvPV_nolen($arg)} }

Remove a star and prepend "OUT" (or

"IN_OUT") for an argument called by reference.

MODULE Foo PACKAGE FOO

intfunc(char * str, OUT int len)

Further Reading

• perlxs• perlxstut• perlxstypemap• perlapi• perlcall• ... and others' XS

code :p

Part IV: From .h files to XS files

We've written XS files by hand so

far.

Of course this can be done with a

tool.

The best-known tool is h2xs.

It's mainly used to expose

constants from a C library.

With the help of C::Scan, you can use

it to expose C functions as well.

$ h2xs -Axan Foo /path/to/header.h

Defaulting to backwards compatibility with perl 5.xx.xIf you intend this module to be compatible with earlier perl versions, please specify a minimum perl version with the -b option.

Writing Foo/ppport.hScanning typemaps... Scanning /home/xxxx/perl5/perlbrew/perls/perl-5.xx.x/lib/5.xx.x/ExtUtils/typemapScanning /path/to/header.h for functions...Scanning /path/to/header.h for typedefs...Writing Foo/lib/Foo.pmWriting Foo/Foo.xsWriting Foo/typemapWriting Foo/Makefile.PLWriting Foo/READMEWriting Foo/t/Foo.tWriting Foo/ChangesWriting Foo/MANIFEST

If the library is stable, this may give you a good starting

point.

However.

The files h2xs generates are rather

old-fashioned.

It's not good at updating only a part

of a distribution, either.

I started writing Convert::H::XS.

https://github.com/charsbar/convert_h_xs

Convert::H::XS takes a C header file, and looks for the minimum

information to write XS components.

use Convert::H::XS;my $converter = Convert::H::XS->new;$converter->process($h_file);

Convert::H::XS also provides methods to write XS

components for convenience.

$converter->write_constants( "xs/constants.inc",);

You can tweak those pieces of information

with a callback.

$converter->write_functions( "xs/functions.inc", sub { my ($type, $name, $args) = @_; ... return ($type, $name, $args);});

Grateful if you help me improve

this at the hackathon.

Part V: CPANTS update

I talked about CPANTS last year.

CPANTS tests what module authors often

forget to test by themselves.

Perl QA Hackathon 2013

in Lancaster

Several metrics are going to be removed.

Several metrics are going to be added.

To add new metrics, we need to

find issues.

Issues are often raised by other QA/toolchain

people.

• Do not ship modules with Module::Install 1.04http://weblog.bulknews.net/post/33907905561/do-not-ship-modules-with-module-install-1-04

• stop shipping MYMETA to CPANhttp://weblog.bulknews.net/post/44251476706/stop-shipping-mymeta-to-cpan

We need to confirm they are

measurable, and widespread.

What can I use to confirm?

• CPANTS databases

• metacpan• CPAN grep

I just wanted something else.

Groonga

• a fulltext search engine• a column store• actively developed (monthly

release)

• client/server (http/gqpt)• C API• Web interface

Groonga::API

https://github.com/charsbar/groonga-api

Also grateful if you help me improve

this at the hackathon.

Thank you!